Developer guide

This guide is for pipeline TDs and core developers who want to modify Spil.

The guide to use Spil is here: user guide.
See also the glossay.

Introduction

Before continuing, it is useful to read the user guide and glossay for an optimal understanding of Spil.

Spil is still in Beta, and this documentation is work in progress. If you are interested in using and modifying Spil, please drop us a line at spil@xeo.info. We will be happy to assist you.

This page is work in progress.

- Configuring spil: existing pipeline / new pipeline
- creating Finders, Getters, Writers

Spil class hierarchy

The Sid class (spil.sid.sid) is the final class in a hierarchy.

The Sids parents are, sorted by specialisation:

BaseSid: implements the factory mechanism
StringSid: String and base attributes
TypedSid: types and the field dictionary
PathSid: path resolving
DataSid: delegation to data access (Finder)

Sid object creation is done by a configurable factory method. The goal is to be able to extend the Sid, and to adapt the factory.

Sid object creation

The implementation and factory mechanism is not yet final.
It might change, without affecting the usage of the API.

BaseSid.__new__ calls a factory as per _factory class attribute.
This factory spil.sid.core.sid_factory.sid_factory() delegates to functions (sid_to_sid, path_to_sid, dict_to_sid), depending on the input parameters. The function returns a Sid object, that is returned by the factory to BaseSid.__new__.

BaseSid.__init__ is empty to avoid initialisation for already initialised instances.

This page is work in progress.

Finders

All “Finder” classes extend Finder. They are typically named “FindIn*”.

FindInPaths: to search the file system
FindInList: to search a list
FindInCache: to search a cache
FindInConstant: to search a list of constants for a given type
FindInGlob: parent class for the glob style searches (Path, List, Cache)
FindInAll: to search other Finders, depending on a configuration
FindInShotgrid: to search Shotgrid

Finders implement the methods find(), do_find(), find_one(), and exists().

Search process

The global search process is as follows.

find() is called with a search sid (string or Sid object). The search sid is most of the times untyped. For example hamlet/a/** matches multiple types. The search sid it “unfolded”, transformed into a list of possible typed search Sids.

Examples of unfolding:

  "hamlet/*/**" is unfolded to: 
  [Sid("asset__file:hamlet/a/*/*/*/*/*/*"),
   Sid("shot__file:hamlet/s/*/*/*/*/*/*"),
   Sid("shot__cache_node_file:hamlet/s/*/*/*/*/*/*/*"),
   ...]

  "hamlet/*/**/maya" is unfolded to:
      [Sid("asset__file:hamlet/a/*/*/*/*/*/ma"),
       Sid("asset__file:hamlet/a/*/*/*/*/*/mb"),
       Sid("shot__file:hamlet/s/*/*/*/*/*/ma"),
       Sid("shot__file:hamlet/s/*/*/*/*/*/mb"),
       ...]

do_find() is called with a list of search sids. It runs the actual search on each typed search sid, and yields the result. (the results are kept in an internal set() to avoid yielded one twice)

Note: find_one() and exists() call find() by default.

Implementing a search

You can implement a Finder to access a data source in your pipeline. Finders are organised in a class hierarchy, and you can override the features as needed.

Not all APIs or data sources are easy to wrap, due to Sids / Finders hierarchical nature.

FindInShotgrid

Have a look at spil_plugins.sg.FindInShotgrid to see how the Shotgrid API is wrapped by a finder.

The powerful design of the Shotgrid API makes it possible to query most types in one single request. The technique is called “field hopping” https://developer.shotgridsoftware.com/python-api/usage_tips.html.

For a relational database, it means querying table joints, or create denormalized views.

This page is work in progress.