Skip to content

Home

Jump to bottom
Stefano Castelluccio edited this page Sep 15, 2026 · 2 revisions

DISCLAIMER

The extension is delivered as-is and the extension is published under GNU LGPL v3 open source license.

1. Description

The Extension should be used when an application under analysis by CAST AIP/Highlight uses one or more NoSQL data stores from Java code (directly, through vendor drivers, or through Spring Data repositories).

The Extension is a Universal Analyzer / JEE Analyzer add-on, and it is intended to extend the technology coverage of CAST AIP for the following technologies:

  • MongoDB (native driver, MongoTemplate/MongoOperations, Spring Data MongoDB repositories, Jongo, Morphia)
  • MarkLogic
  • CouchDB
  • Couchbase (2.x and 3.x, including Spring Data)
  • Redis (Jedis, Lettuce, Redisson, Spring Data Redis)
  • DynamoDB (including Spring Data)
  • CosmosDB (Sync, Async, 4.x, Spring Data)
  • Memcached
  • Elasticsearch (including Spring Data)
  • Apache Lucene
  • Apache Solr
  • Amazon DocumentDB

The Extension adds extra artifacts (databases, collections/indices/keyspaces, connections), extra links (CRUD access links between application code and NoSQL artifacts), and extra quality rules/metrics on top of the standard JEE analysis. It also computes AFP Data Function complexity (DET/RET) for the data artifacts it creates, so that the artifacts can be sized as IFPUG Data Functions.

1.1 Version

The table below indicates the current version of the Extension and the history of the most recent, relevant versions.

  • Version: 2.2.4-beta01002447
  • Build date: 28/05/2026
  • Key Notes (this update): New MongoDB DET/RET evaluation engine in mongodb.py. RET/DET are now computed from the actual @Document-mapped entity structure (or from an optional external JSON schema) instead of being fixed to DET=1/RET=1. Adds support for nested/embedded documents, arrays of documents (RET), scalar arrays, references, polymorphic variants, and an external mongodb_schema*.json override mechanism, with a conservative DET=1/RET=1 fallback when no structure can be resolved.

1.2 Use Cases

The Extension is covering the following use cases:

Extension of quality rules:

  • Technology: MongoDB, MarkLogic, CouchDB, Couchbase, Redis, DynamoDB, CosmosDB, Memcached, Elasticsearch, Apache Lucene/Solr
  • Quality: Best-practice rules on NoSQL usage (e.g. avoid multiple artifacts writing to the same collection, avoid unauthenticated access, avoid explain() in production code, compound index consistency, etc.)

Extension of artifacts:

  • Types: NoSQL databases/keyspaces/indices, NoSQL collections/documents, NoSQL connections, and (new) collection structure artifacts (nested structures and attributes) used to support DET/RET computation for MongoDB collections.

2. Compatibility

2.1 CAST AIP / AIP Console Compatibility

The following table gives the list of CAST AIP configurations where the Extension has been used:

CAST AIP Release Compatible Extension version
8.3.x and above Yes 2.2.x
8.0.x to 8.2.x Yes 1.x to 2.1.x

2.2 Other Technology Compatibility

Requires the standard CAST JEE/Java Analyzer to be installed, since this Extension analyzes Java source code and Spring configuration/annotations.

3. Prerequisites & Installation

3.1 Prerequisites

The following table gives the complete list of technical prerequisites to be met before installing the Extension:

  • An installation of any compatible release of CAST AIP (see table above)
  • The CAST Java/JEE Analyzer must be installed and licensed
  • Java source code must be analyzable (compiles / resolvable classpath) so that Spring Data entity classes, annotations, and field types can be introspected

3.2. Installation Instructions

  • Download the extension from CAST Extension Downloader, or select it in AIP Console
  • Install with CAST Server Manager or AIP Console, following the standard CAST schema/extension installation procedure (Application Analytics platform / Additional Analysis platform installation, then select the extension version to install)

WARNING: All files should be coded in UTF-8, otherwise the extension could fail in detecting violations.

4. How to Use

4.1 Features and Environment

  • If your application contains Java source code using MongoDB, MarkLogic, CouchDB, Couchbase, Redis, DynamoDB, CosmosDB, Memcached, Elasticsearch, Apache Lucene/Solr (directly or via Spring Data), use this Extension to view these object types and their links with the rest of the application
  • This Extension analyzes NoSQL API calls (native driver and Spring Data repositories/templates) and creates the corresponding databases/collections/connections and CRUD links
  • For MongoDB, it additionally resolves the structure (attributes and nested documents) of @Document entities to compute AFP DET/RET on each collection

4.2 Analysis Strategy

Nothing specific beyond a standard Java/JEE analysis. No dedicated Analysis Unit type is required; the Extension hooks into the existing JEE analyzer callbacks.

4.3 MongoDB DET/RET Evaluation (New)

This section documents the new DET (Data Element Type) / RET (Record Element Type) evaluation logic implemented in mongodb.py, used to size CAST_Java_MongoDB_Collection objects as IFPUG-style AFP Data Functions.

4.3.1 Principle

Previously, every MongoDB collection was sized with a fixed DET=1 / RET=1. The new engine instead:

  1. Resolves the Java entity class bound to a collection (via Spring Data repository generics or @Document)
  2. Introspects that entity's field structure (scalars, references, embedded objects, arrays of scalars, arrays of embedded documents)
  3. Optionally merges/overrides that structure with an external JSON schema describing the real MongoDB document shape
  4. Computes DET and RET from the merged structure using AFP counting rules
  5. Falls back to the conservative DET=1 / RET=1 only when no structure at all can be resolved

The computed values are stored as properties on the collection object: CAST_Java_NoSQL_Collection.DET and CAST_Java_NoSQL_Collection.RET (metamodel rid="10001" / rid="10002", see NoSQLForJavaMetaModel.xml).

4.3.2 Entity structure resolution (Java source)

  • extract_type_structure() walks the fields of a @Document-annotated (or repository-generic) entity class using CAST symbol introspection, classifying each field as:
    • scalar (String, numeric wrappers/primitives, Date, ObjectId, etc. — see is_scalar_type())
    • collection/array (List<>, Set<>, Collection<>, ... — see is_collection_type() / extract_generic_type()), recursed into as a nested structure
    • nested object (any other resolvable class), recursed into as an embedded structure
  • _normalize_field_name() prefers the BSON name declared via @BsonProperty/@Field over the Java field name.
  • register_collection_structure() attaches the resolved structure to the collection name in self.collection_structures, merging with (not overwriting) any structure already supplied by an external schema.

4.3.3 Optional external schema override (mongodb_schema*.json)

Because Java field introspection cannot always capture the real, effective MongoDB document shape (dynamic fields, driver-level documents, polymorphism, etc.), the analysis looks for optional external JSON schema files placed under the source tree (searched upward from repository/entity source files, and explicitly under any main_sources folder):

  • Recognized file names: mongodb_schema.json, mongodb_schema1.json … mongodb_schema4.json (and mongodb-, mongo_/mongo-, nosql_/nosql- variants of the same numbered pattern)
  • Discovery/loading is handled by _discover_external_schema_files() / _discover_external_schema_files_for_symbol() and load_external_collection_schemas() / load_external_collection_schemas_for_symbol()
  • Each schema entry is normalized by _normalize_external_structure() into a common structure describing:
    • attributes / fields / detFields — plain DET-eligible fields
    • scalarArrays / arrayScalars — arrays of scalars (counted as a single DET)
    • references / dbRefs / foreignKeys — reference fields
    • technicalFields / ignoredFields — fields excluded from AFP counting
    • nestedStructures / embeddedDocuments / subdocuments / arraysOfDocuments — embedded objects or arrays of documents (RET candidates)
    • polymorphicVariants / variants — alternative shapes for polymorphic documents
    • Collection-level options: classification, countObjectId, fpaCountEmbeddedDocumentsAsRET, fpaBusinessMode, fpaUseExpectedCounts, expectedDET, expectedRET, excludeFromFPA/fpaExcluded, userDistinct/distinctForUser
  • _merge_collection_structure() / _merge_structure_nodes() merge external schema data with Java-derived data without duplicating attributes/nested structures, while letting external collection-level options (e.g. expectedDET/expectedRET) take precedence
  • _force_reload_external_schema_for_collection() is a defensive re-read used when a previously registered structure is empty or under-classified, to avoid losing a valid external schema entry

4.3.4 DET/RET counting rules

Implemented in _field_counts_as_det() and _count_structure():

  • A field counts as one DET if it is a scalar, boolean, reference, or scalar-array field, is not flagged technical, and is not one of the conventionally excluded technical fields (__v, __lastSeen, auto-managed createdAt/updatedAt)
  • The _id field counts as a DET only if significant: true is set, or if countObjectId is enabled at the collection level; otherwise it is excluded
  • Fields/structures flagged excludeFromFPA, fpaExcluded, notBusinessData, or technical: true are excluded from counting (_fpa_is_excluded())
  • The root document itself always counts as RET = 1
  • A nested array of documents (type: array_object) is always counted as an additional RET (recursively, at least 1), and its DET are added to the collection total
  • An embedded document (type: embedded_document / document / object) adds its DET to the total, and only adds a RET if fpaCountEmbeddedDocumentsAsRET/fpaBusinessMode is enabled at the collection level, or the nested node is flagged userDistinct/distinctForUser
  • Polymorphic variants contribute their distinct field names once each to the DET count, and the RET count is raised to at least the number of variants
  • If a schema explicitly provides expectedDET/expectedRET and opts in via fpaUseExpectedCounts, fpaBusinessMode, or fpaCountEmbeddedDocumentsAsRET, those authoritative values are used as-is instead of the computed counts (_compute_collection_det_ret())
  • If the computed structure is empty and would otherwise default to the historical DET=0/RET=1 result, the extension conservatively normalizes it to DET=1 / RET=1 to preserve backward-compatible sizing for collections with no resolvable structure

4.3.5 Materialization

materialize_collection_structure() is called once per collection (from before_parametrisation(), _create_collection_object_for_known_db(), and after_parametrisation() for late-resolved repositories) and:

  1. Resolves/reloads the structure for the collection
  2. Creates lightweight DET-eligible attribute descriptors (safe-mode, no extra custom objects are persisted for individual fields/nested structures — see _create_attribute_object() / _create_structure_object())
  3. Computes the final DET/RET pair via _compute_collection_det_ret()
  4. Saves CAST_Java_NoSQL_Collection.DET and CAST_Java_NoSQL_Collection.RET on the collection object

4.3.6 Spring repository resolution safeguard

Independently of DET/RET, a related safeguard (spring_repository_strict_resolution, _is_confirmed_collection_name(), _extract_confirmed_collection_from_entity(), get_collection_from_entity()) prevents a Spring Data repository from creating a false MongoDB collection out of an unrelated DTO/MVC class: a repository-derived collection name is only materialized when confirmed by @Document(collection=...), by an external schema entry, or by a previously/explicitly detected collection (with a case-insensitive fallback match). Diagnostic tracing for unresolved cases is emitted through _mongo_trace() when mongo_unknown_trace_enabled is set.

4.3.7 Further reference

For the full, versioned technology documentation of this Extension (installation, supported APIs per NoSQL technology, quality rules, release notes), see: https://doc.castsoftware.com/technologies/nosql/extensions/com.castsoftware.nosqljava/2.2/

4.4 Documentation History

No additional information is available beyond the version notes in section 1.1.

5. What Results Can be Expected

Once the analysis/snapshot generation has completed, the following objects, links, and metrics will be available in CAST Enlighten/Highlight.

5.1 Outputs

The Extension produces new artifacts (databases, collections, connections, and — for MongoDB — resolved collection structures used only for DET/RET sizing), parsing Java source code, Spring configuration, and (for MongoDB) optional external mongodb_schema*.json files. It also produces AFP Data Function sizing (DET/RET) on MongoDB collections, and additional quality rule violations related to NoSQL best practices.

5.2 Links

CRUD-style links (insert/select/update/delete) between application code (methods, repository query methods) and the corresponding NoSQL database/collection/connection artifacts.

5.3 Quality Rules

For the detailed list of implemented quality rules per technology, refer to the extension's Quality Model documentation and to the online documentation linked in section 4.3.7.

7. Components Present in the Package

Analysis is accomplished both through standard CAST language/metamodel definition files (see configuration/Languages/NoSQLForJava/NoSQLForJavaMetaModel.xml) together with a Python part covering per-technology detection, link creation, and quality rule tuning — most notably mongodb.py for MongoDB (including the DET/RET evaluation engine described above), and the sibling modules for the other supported NoSQL technologies (cosmosdb.py, couchbase_2x.py, couchbase_3x.py, couchdb.py, dynamodb.py, elasticsearch.py, marklogic.py, memcache.py, redisdb.py, apache_lucene.py, apache_solr.py, etc.).

8. Version of Documentation Used

Version 2.1 of "Extension - Template description page" is used for this document.

Clone this wiki locally