############### Getting Started ############### Installation ------------ Install using ``pip`` from the SKAO central artefact repository:: $ pip install --extra-index-url https://artefact.skao.int/repository/pypi-internal/simple ska-telmodel Simple usage ------------ List data ========= You can now use the command line utility to list default telescope model data:: $ ska-telmodel ls instrument/mccs-configuration/station_export_w2.json instrument/mccs-configuration/antenna_export_w2.json instrument/ska1_low/layout/low-layout.json instrument/ska1_low/layout/data.json instrument/ska1_low/layout/README.md [...] You can achieve the same thing from Python as follows:: from ska_telmodel.data import TMData for key in TMData(): print(key) Retrieve data ============= You can easily retrieve data from the command line as well:: $ ska-telmodel cat instrument/ska1_low/layout/README.md SKA Low layout -------------- [...] Again, the same can be achieved from Python:: from ska_telmodel.data import TMData print(TMData()['instrument/ska1_low/layout/README.md'].get().decode()) For JSON or YAML data, you can especially retrieve it parsed:: print(TMData()['software/tango/ska_wide/Guidelines.yaml'].get_dict()) # -> [{'class': None, [...] Adding Files via the CLI ======================== Files can be added to an existing data source by using the following command structure: .. code-block:: text ska-telmodel upload --repo= Examples of paths that can be used for the ``--repo`` argument: * ``ska-telescope/ska-telmodel`` - references a telescope model data source repository on GitLab **(this is the recomended method)**. This is equivalent to ``ssh://git@gitlab.com:`` (see below) * ``ssh://git@gitlab.com:ska-telescope/ska-telmodel.git`` - references an exact git repo * ``/valid/path/to/tmdata`` - references a local directory to copy data to An example of uploading to a SKA Gitlab repo: .. code-block:: text $ ska-telmodel upload --repo=ska-telescope/sdp/ska-sdp-qa-data-api README.rst path/in/repo/README.rst Add README.rst to git://ska-telescope/sdp/ska-sdp-qa-data-api (path/in/repo/README.rst)? [y|N]: y Local checkout doesn't exist, checking out... Specify branch name to use: nal-1131 Copying file as is Please specify a commit message: NAL-1131 Test upload of new data Created branch: nal-1131, please create relevant merge request. https://gitlab.com/ska-telescope/sdp/ska-sdp-qa-data-api/-/merge_requests/new?merge_request%5Bsource_branch%5D=nal-1131 File added If you are updating an existing file, in a default source. You are be able to shorten the above command to: .. code-block:: text ska-telmodel upload If the ``TMData key`` is the same as ``input path`` then the ``TMData key`` can be omitted: .. code-block:: text ska-telmodel upload For further detail refer to :ref:`CLI Upload`. Adding Files via the Library ============================ To add files via the library (without any interaction from the user). We can use the following code: .. code-block:: python import logging import os import sys import tempfile from pathlib import Path from ska_telmodel.data.large_files import upload_large_file from ska_telmodel.data.new_data_backend import GitBackend logging.basicConfig(level=logging.INFO) ################################################################################ # Config repo = "ska-telescope/ska-telmodel" branch_name = "branch-name" commit_title = "Commit new data" # Set the following to not get asked for each upload # os.environ["CAR_TMDATA_USERNAME"] = "" # os.environ["CAR_TMDATA_PASSWORD"] = "" ################################################################################ print("Repo init...") git_repo = GitBackend(repo=repo) files_to_add_small = [ (Path("layout_1.yaml"), "system/configuration/layout_1.yaml"), (Path("layout_2.yaml"), "system/configuration/layout_2.yaml"), (Path("layout_3.yaml"), "system/configuration/layout_3.yaml"), (Path("layout_4.yaml"), "system/configuration/layout_4.yaml"), (Path("layout_5.yaml"), "system/configuration/layout_5.yaml"), ] files_to_add_large = [ (Path("data_file_1.bin"), "system/data/data_file_1.bin"), (Path("data_file_2.bin"), "system/data/data_file_2.bin"), (Path("data_file_3.bin"), "system/data/data_file_3.bin"), (Path("data_file_4.bin"), "system/data/data_file_4.bin"), (Path("data_file_5.bin"), "system/data/data_file_5.bin"), ] print("Create branch...") try: git_repo.start_transaction(branch_name, create_new_branch=True) except ValueError as err: if str(err) == "Branch Already Exists": print("Branch already exists, try a different branch name") sys.exit(1) else: raise print("Add small files...") for file, key in files_to_add_small: print(f"Adding...{file}") git_repo.add_data(file, key) print("Add large files...") for file, key in files_to_add_large: print(f"Adding...{file}") link_file_contents = upload_large_file(repo, file, key) with tempfile.TemporaryDirectory("ska-telmodel") as dirname: link_file_path = Path(dirname, "temp.link") with open(link_file_path, "w") as link_file: link_file.write(link_file_contents) git_repo.add_data(link_file_path, f"{key}.link") print("Commit...") git_repo.commit(commit_title) print("Push Branch...") git_repo.commit_transaction() After using the above code, a new branch will get pushed to the repo. For you to create a new merge request for. Data sources ------------ Local directory =============== ``ska-telmodel`` has a number of default data sources built-in, which we have been querying above. However, you can override this. For instance, you can use a local directory as a source:: $ mkdir tmdata_demo $ echo Test! > tmdata_demo/test.txt $ ska-telmodel ls --sources=file://tmdata_demo test.txt $ ska-telmodel cat --sources=file://tmdata_demo test.txt Test! This works similarly from Python:: from ska_telmodel.data import TMData tmdata = TMData(['file://tmdata_demo']) print(tmdata['test.txt'].get().decode()) # -> Test! A useful pattern is to use this to create a local copy of telescope model data (see :py:func:`ska_telmodel.cli.CLI.cp`). GitLab & CAR sources ==================== You can also use any GitLab directory as a source:: $ ska-telmodel ls --sources=gitlab://gitlab.com/ska-telescope/ska-telmodel?master#tmdata/software UserWarning: gitlab://gitlab.com/ska-telescope/ska-telmodel?master#tmdata/software not cached in SKA CAR - make sure to add tmdata CI! warnings.warn(warning) tango/dsh/DishManager.yaml tango/ska_wide/Guidelines.yaml tango/ska_wide/SKABaseDevice.yaml tango/ska_wide/SKAMaster.yaml This is useful for development, but as the warning indicates should **not** be used seriously, as GitLab will eventually start blocking these kinds of requests. A better approach is to use the SKAO central artefact repository (CAR) as the source:: $ ska-telmodel ls --sources=car:ska-telmodel?master instrument/ska1_mid/validation/mid-validation-constants.json software/tango/ska_wide/Guidelines.yaml software/tango/ska_wide/SKAMaster.yaml software/tango/ska_wide/SKABaseDevice.yaml software/tango/dsh/DishManager.yaml Using the ``car:`` URI we are now referring to an archive artefact, typically mirroring the contents of a repository (see :ref:`Adding a New GitLab Data Source`). TelModel Database Service ========================= The `Telescope Model Database Service `_ is deployed on-site to serve certain files including the layout file to TMC and SDP. It can be accessed via a ``tmdb:`` source URI:: $ ska-telmodel ls --sources=tmdb://ska-telmodel-db-service.ska1_mid layout/ska1_mid.json layout/ska1_mid/receptors/SKA001.json layout/ska1_mid/receptors/SKA036.json layout/ska1_mid/receptors/SKA063.json layout/ska1_mid/receptors/SKA100.json .. _Dynamic Sources: Dynamic sources =============== The source URIs given above point to dynamic branches (``master``), which means that the results of queries against telescope model data might change. For instance we can go:: $ echo Test! > tmdata_demo/test.txt $ git switch -c my_test_branch $ git add tmdata_demo/test.txt; git commit -m "Telescope model data test"; git push my_test_branch origin $ export SKA_TELMODEL_SOURCES=gitlab://gitlab.com/ska-telescope/ska-telmodel?my_test_branch#tmdata_demo $ ska-telmodel ls test.txt $ ska-telmodel cat test.txt Test! $ echo Test2! > tmdata_demo/test.txt $ git add tmdata_demo/test.txt; git commit -m "Telescope model data test 2"; git push my_test_branch origin $ ska-telmodel cat test.txt Test! $ ska-telmodel cat --update test.txt Test2! Note that the result of our query changed - albeit only after we passed ``--update``, which forced a refresh of the cache. A CAR data source would have the same behaviour if a new package was uploaded by a CI pipeline. In Python we would achieve the equivalent as follows:: from ska_telmodel.data import TMData sources = ['gitlab://gitlab.com/ska-telescope/ska-telmodel?my_test_branch#tmdata_demo'] tmdata = TMData(sources, update=True) print(['test.txt'].get().decode()) Pinning dynamic sources ======================= This dynamic behaviour might be useful in development, but when running code in testing or production, we would like more reproducibility. This is why it is a good idea to "pin" dynamic sources to a specific version. One approach is to refer to a fixed "tag":: $ ska-telmodel --sources=car:ska-telmodel?1.5.0 ls software/tango/ska_wide/Guidelines.yaml software/tango/ska_wide/SKAMaster.yaml software/tango/ska_wide/SKABaseDevice.yaml software/tango/dsh/DishManager.yaml Now we are effectively referring to a "telescope model data release", which is permanently stored in the CAR and will always give the same result. Note that every repository publishing telescope model data might have its own independent version history, and there's especially no connection to the version of the telescope model data library. Another approach is to "pin" sources, which resolves them to hashes:: $ export `ska-telmodel -U pin` Using car:ska-telmodel-data?~9d576afb2f8980bab1fea5d82fa80ddfa91fba21 Using car:ska-telmodel?~719f0146df1de15dfaaa1780847de656ce35c29a Using car:mccs/ska-low-mccs?~6d98ac66b188d9943b2af19e3e5f2f317da384e8 $ echo $SKA_TELMODEL_SOURCES car:ska-telmodel-data?~9d576afb2f8980bab1fea5d82fa80ddfa91fba21,car:ska-telmodel?~719f0146df1de15dfaaa1780847de656ce35c29a,car:mccs/ska-low-mccs?~6d98ac66b188d9943b2af19e3e5f2f317da384e8 In Python we would achieve the same as follows:: from ska_telmodel.data import TMData sources = TMData(update=True).get_sources(pinned=True) print(sources) # -> ['car:ska-telmodel-data?~9d576afb2f8980bab1fea5d82fa80ddfa91fba21', 'car:ska-telmodel?~719f0146df1de15dfaaa1780847de656ce35c29a', 'car:mccs/ska-low-mccs?~6d98ac66b188d9943b2af19e3e5f2f317da384e8'] At this point we would be able to pass ``sources`` to a different component (e.g. a configured sub-system):: # Set telescope model data to use, issue call to other component config['sources'] = tmdata.get_sources(pinned=True) config['layout_key'] = 'instrument/ska1_low/layout/data.json' otherComponent.Command(json.dumps(config)) Now another component (e.g. Tango device) could get the data pointed at as follows: def Commnand(self, config_str): config = json.loads(config_str) tmdata = TMData(config['sources']) layout = tmdata[config['layout_key']] At this point we could be sure that the second piece of code has exactly the same view of telescope model data - regardless of any updates to telescope model data that might have happened in the meantime. Permanently adding or changing files ==================================== In :ref:`Dynamic sources` we used a GitLab source to quickly add a file, but this is not how you would add files to telescope model data permanently. As explained in the last section, to add data long-term we want to make them part of telescope model data "releases" persisted in the central artefact repository (such as ``car:ska-telmodel?1.5.0``). The idea is that **any** SKAO repository can release such telescope model data packages, similar to how any repository can publish (say) Python packages. For instance, the following repositories currently publish telescope model data: * https://gitlab.com/ska-telescope/ska-telmodel - telescope model data directly associated with the telescope model library (data for semantic validation) * https://gitlab.com/ska-telescope/ska-telmodel-data - shared information about the telescope, such as receptor positions * https://gitlab.com/ska-telescope/mccs/ska-low-mccs - MCCS configuration information You can view the information coming from these repositories as usual:: $ ska-telmodel --sources=car:mccs/ska-low-mccs?master ls instrument/mccs-configuration/station_export_w2.json instrument/mccs-configuration/antenna_export_w2.json To add your own information, you need to: 1. Identify the repository where you wish to add information. If your telescope model data does not fit into an existing repository that publishes telescope model data, check :ref:`Adding a New GitLab Data Source` for how to set up a new repository to publish telescope model data. 2. Add the data to the ``tmdata`` folder in the repository, e.g. using a merge request. Make sure you choose a good path within it, because it will be global, see :ref:`Domains`. Once merged, you should be able to see your file using ``ska-telmodel --sources=car:ska-your-repo?main`` (assuming your main branch is called ``main``, otherwise ``master``) 3. Optional: Release your repository (i.e. create a tag) to create a versioned telescope model data package, which can then be accessed using ``ska-telmodel --sources=car:ska-your-repo?a.b.c`` where ``a.b.c`` is the release version. 4. Alternatively new data can be added to an existing repository using the ``upload`` command. More information on this can be found at :ref:`cli`. 5. Using the ``upload`` command, uploading of large files can also be uploaded to the CAR using the ``--car`` or ``--upload-to-car`` flag (or if the file is above the current default of 100MB). An account for the CAR is required for the command. Accounts can be obtained by logging a request with the Systems Team via their portal which can be found `here `_. Further information =================== For more in-depth guides, check :ref:`Usage Guide`. There is also an SKAO Slack channel for helping users and developers of the SKA telescope model - ``#help-telmodel``.