mash-core

Repositories and Storage

Necessary knowledge: Introduction to Mash & Datasets and distributions

Metadata and data can be stored in repositories in Mash. Metadata can be any class that subclasses SemanticModel, while data must be placed in Distributions within Datasets.

Typically the act of storing metadata and data is done through the general mash interface with the following functions (which all use the current active repository by default):

• mash.save(*resources) stores metadata
• mash.write_data(bytes, distribution) adds data to a distribution
• mash.read_data(distribution) reads data from a distribution

Here are some examples:

import json

import mash
from mash import models

dist = models.Distribution(identifier="ex:distribution", mime_type="application/json")
dataset = models.Dataset(
    identifier="ex:dataset", title="A dictionary", description="A simple dictionary of no use outside of this example", distributions=[dist]
)
mash.save(dataset)

# The dataset with containing distributions must be stored in the active repository before any data can be written to the distribution
mash.write_data(json.dumps({"a": "dictionary"}).encode(), dist)

a_dictionary = json.loads(mash.read_data(dist))
print(a_dictionary)

Active repository

At any given time mash has an active directory which will be used when one is not supplied. This is the temp repository by default, which persists data to disk but deletes it when code is finished running. Another repository can be activated using the function mash.use_repository which works the following way:

import mash

# fetch, save, write/read_data will all use the active repository
with mash.use_repository("uploader"):
    dataset = mash.fetch("...identifier")

# use_repository works both as a context manager and normal function
mash.use_repository("dataprod")
mash.save(dataset)

# The above code fetches a dataset from the "uploader" repository and saves it to the "dataprod" repository

Any repository

mash.fetch and mash.read_data both accept the key "any"instead of a repository, in which case they will look for metadata or data in all connected repositories.

Copying and deleting

If you want to copy data and metadata from one repository to another use mash.copy(**resources, destination). By default it copies from the active repository to a specified destination, but you can set the source to be any repository if you want. mash.copy copies all metadata, and will copy any contained distributions and data for dataset resources.

Deletion of data from a repository can be done using mash.delete which as all other functions delete from the active repository by default. It removes both metadata and eventual data from distributions or datasets.

Here are some examples:

import mash

uploader_repo = mash.get_repository("uploader")

# The dataset is fetched from the active repository
dataset = mash.fetch("...identifier")
mash.copy(dataset, uploader_repo)

mash.delete(dataset)
# Now the dataset is deleted from the active repository, but has been copied along with any containing data to the uploader repository

Profile-bound static repositories

Sometimes you have resources associated with a profile, and would like these to always be available where the profile is imported / used. This is solved by creating a static repository for the profile by adding the attribute static_resources:

from mash.core.profiles import MashProfile
from mash import find_semantic_models_in_module
from mash.repositories.mesh.utilities import configure_profile_repository

NemoProfile = MashProfile(
    title="Nemo",
    identifier="https://elbits.no/tjenester/nemo/",
    token="nemo",
    description="A simplified grid model consisting of central referance data made for organizational purposes",
    models=find_semantic_models_in_module(models),
    static_resources=configure_profile_repository(__file__),
)

This repository can be written to as a normal repository, and the resulting files will live within the mash repository where they are stored alongside the code. That way anyone who clones the repository will have access to them.

During runtime mash has its own repository collection that holds all these static repositories, allowing any script to access them through using the static repository (which is a view over the static repository collection.)

Mash will need to be told to activate the relevant static resources, either by using the profile as a context manager - or by sending it into globally_use_profile.

Recommended next read:

• Intro to profiles