.. _Usage Guide: Usage Guide =========== .. _Domains: Data ---- Telescope model data is stored as `objects` identified as `keys`. Each key takes the following form:: [domain]/([sub-domain]/)*[name].[type] Where * ``[domain]/`` specifies the coarse top-level telescope model data domain. * ``[sub-domain]/`` provides further hierarchical data sub-categories. * ``[name]`` associates a name with the telescope model data. * ``[type]`` identifies the file type, which is used to identify the kind of file contents. The library currently supports ``json`` and ``yaml``. Names should be chosen for being self-describing and stable long-term. Top-level domains:: environment/... # Environment telescopes are deployed in instrument/... # Telescopes and their equipment software/... # Software deployed to the telescopes To retrieve a particular piece of data from the telescope model, simply construct an :py:class:`ska_telmodel.data.TMData` object and use the ``[]`` operator to access:: from ska_telmodel.data import TMData tmdata = TMData() print(tmdata['instrument/ska1_low/layout/low-layout.json'].get_dict()) This works because the telescope model comes with a number of default sources that will be checked for matching telescope model data. :py:meth:`ska_telmodel.data.TMObject.get_dict` automatically parses and converts JSON and YAML documents, but you can also get the raw data using :py:meth:`ska_telmodel.data.TMObject.get`, or open or copy the contents as a file using :py:meth:`ska_telmodel.data.TMObject.open` or :py:meth:`ska_telmodel.data.TMObject.copy` respectively. To get an idea what is contained in a particular part of the telescope model data tree, simply iterate over it (equivalent to :py:meth:`ska_telmodel.cli.CLI.ls`):: from ska_telmodel.data import TMData tmdata = TMData() for key in tmdata['instrument']: print(key) Logically, the ``[]`` operator constructs a subset of all telescope model data. If the key is a valid object name (i.e. has an extension, so contains a ``'.'``) this subset is assumed to contain only a single object, and the ``[]`` operator will return a :py:class:`ska_telmodel.data.TMObject` instance. .. _Data Sources: Data Sources ------------ Telescope model data can be retrieved from a list of sources, which can be specified to the :py:class:`ska_telmodel.data.TMData` constructor, using the ``SKA_TELMODEL_SOURCES`` environment variable or left to in-built :py:const:`ska_telmodel.data.sources.DEFAULT_SOURCES`. Each source is represented as a URI that specifies the source of truth for some portion of telescope model data. The following telescope model data backends are currently supported: * ``mem://?[key1]=[value1]&[key2]=[value2]`` (see :py:class:`ska_telmodel.data.backend.MemoryBackend`) * ``file://[absolute path]`` (see :py:class:`ska_telmodel.data.backend.FilesystemBackend`) * ``gitlab://[gitlab server]/[project name]?[branch]#[directory]`` (see :py:class:`ska_telmodel.data.backend.GitlabBackend`) * ``car://[gitlab server]/[project name]?[branch]#[directory]`` (see :py:class:`ska_telmodel.data.backend.CARBackend`) * ``tmdb://[telmodel DB server]?[rev=timestamp]#[directory]`` (see :py:class:`ska_telmodel.data.backend.DatabaseBackend`) The simplest example would be to utilise :py:class:`ska_telmodel.data.backend.MemoryBackend` to set a key directly:: from ska_telmodel.data import TMData tmdata = TMData(['mem://?test.txt=test_data']) print(tmdata['test.txt']) # -> b"test_data\n" This can also be configured using environment variables:: import os from ska_telmodel.data import TMData os.environ['SKA_TELMODEL_SOURCES'] = 'mem://?test.txt=test_data' tmdata = TMData() print(tmdata['test.txt']) # -> b"test_data\n" You would typically do this from outside your program, see the documentation for :py:meth:`ska_telmodel.cli.CLI.pin` and :py:meth:`ska_telmodel.cli.CLI.cp` for examples. A more complex example would be to retrieve data from Gitlab using :py:class:`ska_telmodel.data.backend.GitlabBackend`:: from ska_telmodel.data import TMData gl_uri = 'car://gitlab.com/ska-telescope/ska-telmodel?master#tmdata' tmdata = TMData([gl_uri]) print(tmdata['software/tango/dsh/DishManager.yaml']) This will retrieve data directly from the telescope model library repository. Note that external telescope model data sources using :py:class:`ska_telmodel.data.backend.GitlabBackend` or :py:class:`ska_telmodel.data.backend.CARBackend` will cache data locally in order to prevent repeated requests to servers. This means that if we reference a Gitlab branch (like ``master`` in the example), the telescope model data in the cache might go out of sync with the server. This is intentional, as it means that we provide a consistent view of telescope model data as long as possible. It is generally best to use "pinned" sources (see :py:meth:`ska_telmodel.cli.CLI.pin`), but in day-to-day usage, you can simply use the ``-U`` flag as documented in :ref:`cli` or (less preferably) the ``update`` option to :py:class:`ska_telmodel.data.TMData` to occassionally refresh the cache as needed. The library will occassionally check for and warn about stale caches. .. _Adding a New Gitlab Data Source: Adding a New GitLab Data Source ------------------------------- If you want others to be able to view data in your GitLab repository using :py:class:`ska_telmodel.data.backend.GitlabBackend` or :py:class:`ska_telmodel.data.backend.CARBackend`, first you will need to place the data you wish to export in a top level dictionary in your repo named ``tmdata``. For example:: /tmdata/instrument/mccs_configuration/config_file_low.json /tmdata/instrument/mccs_configuration/config_file_mid.json Important to note: * Try to use a directory structure that is compatible with domains (see above) and is reasonably likely to remain stable. * Currently only ``.json`` and ``.yaml`` files are accepted, and you should have schemas associated with them. Next add telescope model data support to your top-level ``Makefile`` as documented in https://developer.skao.int/projects/ska-cicd-makefile/en/latest/README.html : :: include .make/tmdata.mk At this point you should be able to verify that ``make tmdata-package`` will result in both a ``tmtree.json`` and a ``tmdata.tar.gz`` file getting created in ``build/tmdata``. Next add the packaging and publishing stage to your GitLab pipeline by adding the following lines to the ``.gitlab-ci.yml`` file as documented in https://developer.skao.int/projects/templates-repository/en/latest/README.html : :: - project: 'ska-telescope/templates-repository' file: 'gitlab-ci/includes/tmdata.gitlab-ci.yml' Now once you merge these changes into the ``main`` branch, others will be able to access this data by specifying your repository as the source:: $ ska-telmodel --sources=car:mccs/ska-low-mccs?main ls instrument/mccs_configuration/config_file_low.json instrument/mccs_configuration/config_file_mid.json Branches other than ``main`` will also work, just adjust the URL accordingly. However by default the GitLab pipeline will only upload the ``TMData`` package to the artefact repository on the main branch as well as tags. If you want the data to be accessible without passing command line parameters, make a merge request to the ska-telmodel repository ( https://gitlab.com/ska-telescope/ska-telmodel ) that adds your repository address the ``src/ska_telmodel/data/source.py`` file. This makes your telescope model data available "globally":: $ ska_telmodel ls [...] instrument/mccs_configuration/config_file_low.json instrument/mccs_configuration/config_file_mid.json Schemas ------- Schemas check JSON-like objects for conformance, e.g. nested dictionaries containing primitives and lists. They especially have a JSON schema representation - though :py:meth:`ska_telmodel.schema.validate` will generally implement more thorough checks. All schemas are identified by a URI of the form:: https://schema.skao.int/ska-[subsystem]-[interface]/[major].[minor] The entire URI should be lower-case alphanumerical. The ``[subsystem]`` identifies the leading party for maintaining the schema, and ``[interface]`` the concrete interface implemented. Depending on context, this might either be data produced or consumed by the sub-system in question. Versioning should follow semantic versions: Changes in minor version indicate backwards-compatible changes such as adding new fields or otherwise introducing additional accepted schemas. Changes that break backwards compatibility should change the major version. You can use the URIs with :py:meth:`ska_telmodel.schema.validate` to validate data:: from ska_telmodel.data import TMData from ska_telmodel.schema import validate uri = "https://schema.skao.int/ska-telmodel-layout-location/0.0" layout_dict = TMData()['instrument/ska1_low/layout/low-layout.json'].get_dict() validate(uri, layout_dict) Furthermore you can use :py:meth:`ska_telmodel.schema.example_by_uri` to retrieve examples of certain schemas (which are replicated in the schema section of this documentation). Schema validation using JSON schemas ------------------------------------ The schema validation documented above works only for schemas (and schema versions) that are supported by the used version of the telescope model library. However, these schemas are also make public on the Internet, and therefore can can be retrieved as JSON schemas. This allows "dynamically" checking against newer schemas, which is useful as a fallback. To use this, set ``jsonschema_fallback=True`` parameter for the :py:meth:`ska_telmodel.schema.validate` function:: from ska_telmodel.data import TMData from ska_telmodel.schema import validate uri = "https://schema.skao.int/ska-telmodel-layout-location/0.0" layout_dict = TMData()['instrument/ska1_low/layout/low-layout.json'].get_dict() validate(uri, layout_dict, jsonschema_fallback=True) Keep in mind that: * JSON validation is not quite the same as the default validation method performed by the telescope model library (it is generally more permissive) * The HTTP request can fail, so this method is less reliable overall. * However, this function will cache schemas, so repeated calls are faster and safe (see :py:meth:`ska_telmodel.schema._validate_schema_using_document_url`)