tqa
Aspect (of a fact in an XBRL instance).
Aspect (of a fact in an XBRL instance).
See http://www.xbrl.org/specification/variables/REC-2009-06-22/variables-REC-2009-06-22.html#sec-aspects.
Aspect model.
BigDecimal value or XPath expression.
EName value or XPath expression.
EName value or XPath expression. If it is an ENameExpr, the XPath expression is expected to resolve to schema type xs:QName. Hence, a very simple ENameExpr would syntactically look like a lexical QName literal (and not like an EName in James Clark notation).
An XPath expression in a Scope.
An XPath expression in a Scope. Typically this object originates from an XPath expression in an XBRL formula or table linkbase, where the Scope is taken from the XML element scope.
It is likely that the real Scope used to evaluate the XPath expression is not exactly the same Scope as stored in this object. For example, the default namespace for evaluating the XPath expression should probably not be the one stored in this object, if any.
This class is in the "tqa" root package and not in the "xpath" sub-package. The reason is that this class represents data that requires no XPath evaluator whatsoever, whereas the "xpath" sub-package offers an XPath evaluator abstraction. As a consequence a dependency on the "xpath" sub-package is only needed where XPath evaluation is done.
String value or XPath expression.
A collection of substitution groups, typically extracted from a taxonomy.
A collection of substitution groups, typically extracted from a taxonomy. It maps substitution groups to their own substitution groups, if any. Well-known substitution groups such as xbrli:item, xbrli:tuple, xbrldt:hypercubeItem and xbrldt:dimensionItem must not occur as keys in the mapping, but typically do occur as mapped values.
This class is essential for providing the necessary context in order to determine whether a global element declaration is a concept declaration, and, if so, what kind of concept declaration.
Cycles are not allowed when following mappings, but this is not checked.
Value of some type or XPath expression that must evaluate to the same type.
A unique identifier of an XML fragment in a Taxonomy.
A unique identifier of an XML fragment in a Taxonomy. It is made up by the document URI and the Path within that document.
A parsed xsi:schemaLocation attribute value, as a mapping from namespace names to document URIs.
A parsed xsi:schemaLocation attribute value, as a mapping from namespace names to document URIs.
It can not contain any duplicate namespace names, or else an exception is thrown during construction.
Well-known ENames.
Well-known Namespaces.
Yaidom BackingElemApi implementations that can be used as underlying element implementation in
the TQA taxonomy DOM.
Classes that are common to the DOM and model packages, typically "enumerations".
Some console programs.
Type-safe XBRL taxonomy DOM API.
Type-safe XBRL taxonomy DOM API. This contains DOM-like elements in taxonomy documents. It offers the yaidom query API
and more, and wraps an underlying element that itself offers the yaidom query API, whatever the underlying element
implementation. This package does not offer any taxonomy abstractions that cross any document boundaries, except
for the light-weight TaxonomyBase abstraction at the type-safe DOM level.
This package mainly contains:
This package has no knowledge about and dependency on XPath processing.
Suppose we have an eu.cdevreeze.tqa.dom.XsdSchema called schema. Then we can find all global element declarations in
this schema as follows:
import scala.reflect.classTag import eu.cdevreeze.tqa.ENames import eu.cdevreeze.tqa.dom.GlobalElementDeclaration // Low level yaidom query, returning the result XML elements as TaxonomyElem elements val globalElemDecls1 = schema.filterChildElems(_.resolvedName == ENames.XsElementEName) // Higher level yaidom query, querying for the type GlobalElementDeclaration // Prefer this to the lower level yaidom query above val globalElemDecls2 = schema.findAllChildElemsOfType(classTag[GlobalElementDeclaration]) // The following query would have given the same result, because all global element declarations // are child elements of the schema root. Instead of child elements, we now query for all // descendant-or-self elements that are global element declarations val globalElemDecls3 = schema.findAllElemsOrSelfOfType(classTag[GlobalElementDeclaration]) // We can query the schema for global element declarations directly, so let's do that val globalElemDecls4 = schema.findAllGlobalElementDeclarations
Global element declarations in isolation do not know if they are (item or tuple) concept declarations. In order to
turn them into concept declarations, we need a SubstitutionGroupMap as context. For example:
import eu.cdevreeze.tqa.dom.ConceptDeclaration // One-time creation of a ConceptDeclaration builder val conceptDeclBuilder = new ConceptDeclaration.Builder(substitutionGroupMap) val globalElemDecls = schema.findAllGlobalElementDeclarations val conceptDecls = globalElemDecls.flatMap(decl => conceptDeclBuilder.optConceptDeclaration(decl))
Most TQA client code does not start with this package, however, but works with entire taxonomies instead of individual type-safe DOM trees, and with relationships instead of the underlying XLink arcs.
See the remarks on leniency for type TaxonomyElem and its sub-types. This type-safe XBRL DOM model has been
designed to be very lenient on instantiation of the model. Therefore this TQA type-safe DOM model can also
be used for validating potentially erroneous taxonomy documents.
On the other hand, if the instantiated model cannot be trusted to be schema-valid, one should be careful in choosing the API calls that can safely be made on schema-invalid taxonomy content. Yaidom API level query methods that return collections or options are typically safe to use on potentially schema-invalid taxonomy content.
To get the model right, there are many sources to look at for inspiration. First of all, for schema content there are APIs like the Xerces schema API (https://xerces.apache.org/xerces2-j/javadocs/xs/org/apache/xerces/xs/XSModel.html). Also have a look at http://www.datypic.com/sc/xsd/s-xmlschema.xsd.html for the schema of XML Schema itself. Moreover, there are many XBRL APIs that model (instance and) taxonomy data.
On the other hand, this API (and the entirety of TQA) has its own design. Briefly, it starts bottom-up with yaidom, and gradually offers higher level (partial) abstractions on top of that. It does not hide yaidom, however.
TODO Support for built-in schema types, and built-in XBRL types.
Model of XBRL instances, as "yaidom dialect".
Traits offering parts of a taxonomy query API.
Traits offering parts of a taxonomy query API. They can be assembled into "taxonomy classes". There are purely abstract query API traits, and partial implementations of those traits.
Examples of such traits are traits for querying schema content, for querying inter-concept relationships,
for querying dimensional relationships in particular, etc. These traits combined form the
eu.cdevreeze.tqa.queryapi.TaxonomyApi query API. The partial implementations combined form the
eu.cdevreeze.tqa.queryapi.TaxonomyLike trait, which implements most of the TaxonomyApi query API.
Most query API methods are quite forgiving when the taxonomy is incomplete or incorrect. They just return the queried data to the extent that it is found. Only the getXXX methods that expect precisely one result will throw an exception if no (single) result is found.
Ideally, the taxonomy query API is very easy to use for XBRL taxonomy scripting tasks in a Scala REPL. It must also be easy to mix taxonomy query API traits, and to compose taxonomy implementations that know about specific relationship types (such as in formulas or tables), or that store specific data that is queried quite often.
TQA (except the "richtaxonomy" namespace) has no knowledge about XPath, so any XPath in the taxonomy is just text, as far as TQA is concerned.
This package unidirectionally depends on the eu.cdevreeze.tqa.relationship and eu.cdevreeze.tqa.dom packages.
In the following examples, assume that we have a taxonomy of type eu.cdevreeze.tqa.queryapi.TaxonomyApi,
for example a eu.cdevreeze.tqa.taxonomy.BasicTaxonomy. It may or may not be closed under "DTS discovery rules".
The examples show how the taxonomy query API, along with the types in packages eu.cdevreeze.tqa.relationship and
eu.cdevreeze.tqa.dom can be used to query XBRL taxonomies.
Suppose we want to query the taxonomy for all English verbose concept labels, grouped by the concept target EName.
Note that the "target EName" of a concept declaration is the name attribute along with the target namespace of the
schema, as yaidom EName. Here is how we can get the concept labels:
import scala.reflect.classTag import eu.cdevreeze.yaidom.core.EName import eu.cdevreeze.tqa.relationship.ConceptLabelRelationship val concepts: Set[EName] = taxonomy.findAllConceptDeclarations.map(_.targetEName).toSet val conceptLabelRelationshipsByConceptEName = (concepts.toIndexedSeq map { conceptEName => val conceptLabelRels = taxonomy.filterOutgoingConceptLabelRelationships(conceptEName) { rel => rel.language == "en" && rel.resourceRole == "http://www.xbrl.org/2003/role/verboseLabel" } (conceptEName -> conceptLabelRels) }).toMap val verboseEnConceptLabels: Map[EName, Set[String]] = conceptLabelRelationshipsByConceptEName mapValues { rels => rels.map(_.labelText).toSet }
In the example above, each concept should have at most one English verbose label, unless relationship prohibition/overriding is used. Validating this is an exercise for the reader, as they say. Note that in the example above, we mainly used the following knowledge about XBRL taxonomies: we get concept-labels by querying for concept-label relationships, which are standard relationships.
Now suppose we want to find all English terse concept labels, grouped by the concept target EName, but only for
concrete primary items. So we exclude abstract concepts, tuples, hypercubes and dimensions. Here is how:
val concepts: Set[EName] = taxonomy.filterPrimaryItemDeclarations(_.isConcrete).map(_.targetEName).toSet val conceptLabelRelationshipsByConceptEName = (concepts.toIndexedSeq map { conceptEName => val conceptLabelRels = taxonomy.filterOutgoingConceptLabelRelationships(conceptEName) { rel => rel.language == "en" && rel.resourceRole == "http://www.xbrl.org/2003/role/terseLabel" } (conceptEName -> conceptLabelRels) }).toMap val terseEnConceptLabels: Map[EName, Set[String]] = conceptLabelRelationshipsByConceptEName mapValues { rels => rels.map(_.labelText).toSet }
To simulate how TQA retrieves concrete primary item declarations, we could write more verbosely:
import eu.cdevreeze.tqa.dom.ConceptDeclaration import eu.cdevreeze.tqa.dom.PrimaryItemDeclaration val substitutionGroupMap = taxonomy.substitutionGroupMap val conceptDeclarationBuilder = new ConceptDeclaration.Builder(substitutionGroupMap) val concepts: Set[EName] = taxonomy.filterGlobalElementDeclarations(_.isConcrete). flatMap(decl => conceptDeclarationBuilder.optConceptDeclaration(decl)). collect({ case decl: PrimaryItemDeclaration => decl }).map(_.targetEName).toSet
To simulate how TQA filters the concept label relationships we are interested in, we could write more verbosely:
import eu.cdevreeze.tqa.ENames // Falling back to more general method filterOutgoingStandardRelationshipsOfType val conceptLabelRelationshipsByConceptEName = (concepts.toIndexedSeq map { conceptEName => val conceptLabelRels = taxonomy.filterOutgoingStandardRelationshipsOfType( conceptEName, classTag[ConceptLabelRelationship]) { rel => rel.resolvedTo.resolvedElem.attribute(ENames.XmlLangEName) == "en" && rel.resolvedTo.resolvedElem.attributeOption(ENames.XLinkRoleEName).contains("http://www.xbrl.org/2003/role/terseLabel") } (conceptEName -> conceptLabelRels) }).toMap val terseEnConceptLabels: Map[EName, Set[String]] = conceptLabelRelationshipsByConceptEName mapValues { rels => rels.map(_.resolvedTo.resolvedElem.text).toSet }
Suppose we want to query the taxonomy for the parent-child presentation hierarchies in some custom ELR (extended link role). Note that the result can come from multiple linkbase documents, and TQA takes care of that if we query for parent-child relationships instead of querying for the underlying XLink presentation arcs. Here is how we get the top 2 levels:
import scala.collection.immutable import eu.cdevreeze.tqa.relationship.ParentChildRelationship val parentChildRelationships = taxonomy.filterParentChildRelationships(_.elr == customElr) val topLevelConcepts: Set[EName] = parentChildRelationships.map(_.sourceConceptEName).toSet.diff( parentChildRelationships.map(_.targetConceptEName).toSet) val topLevelParentChildren: Map[EName, immutable.IndexedSeq[EName]] = (topLevelConcepts.toIndexedSeq map { conceptEName => val parentChildren = taxonomy.filterOutgoingParentChildRelationships(conceptEName)(_.elr == customElr) val childENames = parentChildren.sortBy(_.order).map(_.targetConceptEName) (conceptEName -> childENames) }).toMap
These examples only scratch the surface of what is possible. Dimensional relationship queries are typically more interesting than the examples above, for example. See eu.cdevreeze.tqa.queryapi.DimensionalRelationshipContainerApi for the dimensional query API that is part of eu.cdevreeze.tqa.queryapi.TaxonomyApi.
The performance characteristics of the eu.cdevreeze.tqa.queryapi.TaxonomyApi trait and its implementations partially depend on the concrete "taxonomy" class used. Still we can say in general that:
In other words, in "inner loops", do not query for taxonomy content other than querying based on specific concept ENames! Note that in the examples above, we started with a slow query, and used fast queries based on concept ENames after that. Keep in mind that in taxonomies with millions of relationships the slow queries may have to process collections of all those relationships.
This package contains relationships.
This package contains relationships. Relationships are like their underlying arcs, but resolving the locators. Note that an arc may represent more than 1 relationship.
This package mainly contains:
Relationship factories extract relationships from a eu.cdevreeze.tqa.dom.TaxonomyBase. They can be used directly, but typically they are used implicitly when creating a eu.cdevreeze.tqa.taxonomy.BasicTaxonomy.
This package has no knowledge about and dependency on XPath processing.
For the usage of this API, see packages eu.cdevreeze.tqa.queryapi and eu.cdevreeze.tqa.taxonomy.
This package unidirectionally depends on the eu.cdevreeze.tqa.dom package.
Taxonomy classes, containing type-safe DOM trees, and mixing in taxonomy query API traits.
Taxonomy classes, containing type-safe DOM trees, and mixing in taxonomy query API traits. In particular, the eu.cdevreeze.tqa.queryapi.TaxonomyApi trait is mixed in as taxonomy query API. See package eu.cdevreeze.tqa.queryapi for more information about how to query XBRL taxonomy content.
The term taxonomy is used here in a very general sense, namely as a collection of taxonomy documents.
Various scenarios are supported. Taxonomies that are not closed (and not validated in any way) must be supported in order for TQA to be useful for taxonomy validation. Closed taxonomies are supported for reliable taxonomy querying. Taxonomies that model networks of relationships are also supported. Specific taxonomies knowing about formulas and/or tables are also supported. Extension taxonomies are also supported.
Some important operations on taxonomies are prohibition/overriding resolution (to find networks of relationships), combining taxonomies (for building extension taxonomies, for example), filtering relationships (to ignore relationships that we are not interested in).
Each taxonomy class has at least the following state (directly or indirectly): a collection of taxonomy DOM root elements, and a collection of relationships. The underlying arcs, locators and resources of those relationships must exist in the collection of taxonomy DOM trees, or else the taxonomy is corrupt.
TQA has no knowledge about XPath, so any XPath in taxonomies is just text, as far as TQA is concerned.
This package unidirectionally depends on the eu.cdevreeze.tqa.queryapi, eu.cdevreeze.tqa.relationship and eu.cdevreeze.tqa.dom packages.
TQA bootstrapping.
TQA bootstrapping.
First of all, bootstrapping needs an URI converter, that converts original (http or https) URIs to local file system URIs. Next, we need that URI converter and a Saxon or yaidom document parser to configure a DocumentBuilder. Next we need a discovery strategy for obtaining the root elements of the taxonomy. This is typically DTS discovery (the details of which can be somewhat tweaked). Finally we need a RelationshipFactory (and maybe an arc filter) to create a BasicTaxonomy.
Once a BasicTaxonomy is created, it can be used as basis for wrapper taxonomy objects that know about networks of relationships, tables/formulas, etc.
A typical URI converter originates from a catalog file, as restricted in the Taxonomy Packages specification. Typical discovery strategies ("taxonomy root element collectors") use taxonomy packages, and typically they are backed by thread-safe (Google Guava) root element caches.
The bootstrapping process is inherently flexible in supporting the loading of more or less broken taxonomies. For example, backing element builders can be made to post-process broken input before the taxonomy DOM is instantiated. As another example, relationships resolution can be as lenient as desired.
This package unidirectionally depends on the eu.cdevreeze.tqa.taxonomy, eu.cdevreeze.tqa.queryapi, eu.cdevreeze.tqa.relationship and eu.cdevreeze.tqa.dom packages.
Purely abstract API of XLink content, as used in XBRL.
Purely abstract API of XLink content, as used in XBRL. Different type-safe DOM abstractions share this API for XLink content.
See the schema xl-2003-12-31.xsd (and the imported xlink-2003-12-31.xsd). For the relevant part of the Core XBRL Specification, see http://www.xbrl.org/Specification/XBRL-2.1/REC-2003-12-31/XBRL-2.1-REC-2003-12-31+corrected-errata-2013-02-20.html#_3.5.
Note that this package contains no linkbase type. Linkbases are only supported by the "taxonomy DOM" package.
XPath evaluation abstraction.
XPath evaluation abstraction. It is needed for processing formula and table link content. It is as technology-agnostic as possible.
This package is not used in most of TQA. It is only used in the "richtaxonomy" namespace, so most of TQA has no knowledge about and dependency on XPath processing.
XPath-aware taxonomy querying support.
XPath-aware taxonomy querying support. In TQA no taxonomy XPath awareness is anywhere outside of this namespace. This implies that all of TQA outside of this namespace (and the xpath namespace) can be used without the need for any XPath processor. Everything inside this namespace does need an XPath processor, though.
Dependency on XPath processing for taxonomy querying and processing is not without its costs:
Root package of the Taxonomy Query API. This package itself contains commonly used data classes and many constants for namespaces and expanded names.
There are 3 layers in TQA. The lowest layer is the type-safe taxonomy DOM model. It uses yaidom (https://github.com/dvreeze/yaidom) for its "XML dialect support", where the XML dialect is XBRL taxonomy data. It knows only about individual DOM trees.
On top of the type-safe DOM layer is the relationship layer. It resolves the arcs of the DOM layer as relationships. Of course, to resolve arcs we need other documents as context.
On top of the relationship layer is the taxonomy query API layer. It uses the underlying layers to offer a query API in which taxonomy elements (such as concept declarations) and relationships can easily be queried.
It is important to note that higher layers do not abstract away lower layers. Think of the layering more in terms of dependencies. The "taxonomy DOM" layer depends only on yaidom, the relationship layer only depends on the taxonomy DOM layer, and the query API layer depends on both the taxonomy DOM and relationship layers. On the other hand, relationships are indeed abstractions of the underlying XLink arcs and locators/resources.
These 3 layers are as follows in terms of packages, from low to high:
Usage
How do we query XBRL taxonomies with TQA? For some examples, see package eu.cdevreeze.tqa.queryapi.
It should be noted that typical queries involve all 3 packages eu.cdevreeze.tqa.queryapi, eu.cdevreeze.tqa.relationship and eu.cdevreeze.tqa.dom.
It should also be noted that TQA is deeply rooted in the yaidom XML query API (https://github.com/dvreeze/yaidom). This is true internally and externally. Internally TQA has been built in a bottom-up manner on top of yaidom, and this is in particular visible in the internals of the TQA DOM package. Externally many TQA query methods remind of yaidom query methods, but taking and returning type-safe TQA DOM taxonomy content instead of arbitrary XML content. In other words, knowing yaidom helps in getting to know TQA. Of course, knowing the basics of XBRL taxonomies and XBRL dimensions also helps in getting to know TQA.
For all TQA queries, it is possible to write an equivalent lower level query, possibly even a low level yaidom query. Doing so may be time-consuming, but it can also be a good learning experience if one intends to use TQA extensively.
TQA tries to follow the XBRL Core and Dimensions specifications. For example, TQA knows about networks of relationships, XPointer in an XBRL context, embedded linkbases etc.
TQA feels like an API designed in a bottom-up manner on top of yaidom, and that is indeed what it is. So, although TQA can help in learning more about XBRL taxonomies, it is not an API for users who have no knowledge about XBRL. From the lower level DOM-like operations to the higher level dimensional bulk query methods, TQA is useful only for users who know what data they are looking for in a taxonomy.
See package eu.cdevreeze.tqa.queryapi for some examples, and navigate to the eu.cdevreeze.tqa.relationship and eu.cdevreeze.tqa.dom packages for specific relationships and type-safe taxonomy DOM content, respectively.
Backing XML DOM implementations
TQA is flexible in the DOM implementation used. It is pluggable, as long as it offers a yaidom
BackingElemApifacade and we have a eu.cdevreeze.tqa.backingelem.DocumentBuilder for it.Out of the box, 2 backing DOM implementations are available:
BackingElemApiimplementation wrapping SaxonNodeInfoobjects, and typically implemented using Saxon tiny treesBackingElemApinative implementation using yaidom "indexed" elementsThe Saxon (9.7) wrapper implementation, especially when tiny trees are used under the hood, is efficient in space and time, so it should be preferred for production code. The native yaidom implementation is less efficient, in particular in memory footprint, but it is easy to debug and therefore a nice choice in test code. Whichever the choice of underlying DOM implementation, most code using TQA is completely unaffected by it.