Meta#Grid

A blog about meta data, BI and data warehousing

Features

Basics

Business rules
  • Keyfigures / Measures: Definition of calculations, formulas, aggregations
  • Attributes: Write down deep hidden knowledge about extracting information of attributes.
    Example: If an product number has a leading „s“ it is only sold via online shop. Define (pseudo-) code to extend your business knowledge to gather information out of your data
Technical Information
  • Sourcesystems: Create a list of sourcesystems (on-prem or cloud) you already use or want to use to extract data and make use of it in a data warehouse or business intelligence system (this may also be cloud references like Google Analytics etc.)
  • Databases: List all databases (relational, big data, graph, …) or webservices that provide or holds data. Also your data warehouse or big data/data lake should be in this list
  • Tables and fields: From relational tables in a database to simple flatfiles; all fields which are avaiable or shall be avaiable in the future are defined here.
    GDPR relavant data are flagged.
  • Tools: If the infrastructure gets bigger, different tools of different vendores are in use. meta#grid holds a list of them to be referenced in different items.
Presentation and Data Movement
  • Data Transfer Processes: All data which are transferred, extracted or modified is listed at one place. ETL jobs, stored procedures, scripts (PowerShell, Bash, Python, Perl, …) are collected in this group
  • Data Delivery Objects: All reports, lists, exports (XLSX, CSV, …), Dashboards, KPI Cockpits are listed in this group to have a summary of all presentations of data
  • Schedulings: In most of the cases jobs (ETL, report bursting, …) are configured to run unattended. The schedules can be defined at this place (cron jobs entries, Apache Airflow items, Microsoft SQL Server Agent Jobs, …)
Relations
  • Mappings: All items can be linked to each other to get a connection.
  • Dependencies: Define parent child relations
  • Business glossary: Define and share vocabulary used in your organisation
  • Brackets: Define synonyms of field names or measures with different spelling (via patterns) or group context based fields/information together (e. g. GDPR relevant fields in your data warehouse)
  • URLs: Define URLs to locate to external items like intranet sites (Microsoft SharePoint, Confluence, Wiki), tickets (JIRA, GitHub Issues) or files on a shared drive
Organisation

Contacts and Contagsgroups: Add contacts of your company or external member which are relevant.

Examples:

  • [Contact Group] Sales Department which are using several data marts and reports
  • [Contact] Data owner of a marketing report in sweden
  • [Contact] ETL developer
  • [Contact] Business analyst in USA which make use of the controlling data of several OLAP cubes

Good to know

  • Multi-User, multi-roles (need to know security implementation)
  • Perspectives: Multi-Language and Speech (you can define your own wording)
  • Multi client and project independent data
  • Automatic history of changes (nothing is lost)
  • Robust relational database to hold all the data
  • Export data as CSV, XML or JSON files
  • Create, edit, comment everything
  • Link each item (map object to object)
  • Bulk load from source systems (extract metadata from databases and Business Intelligence software)
  • Tag every item with global, projectspecific and personal keywords
  • Brackets: Create synonyms or aliases for objects with different spelling

Tagging

Need to know / user roles

Perspectives

meta#grid is targeted to be general purpose. This means, that the items have a general naming convention.

Example: ETL items are grouped as „Data transfer processes“. If your users need a more specific wording you can change the perspective to language-tool-translation like „Informatica español“ or „Microsoft SSIS Deutsch“ to see spanish or german descriptions.

This can be customized to be the speech of your line-of-business from consumer to developer.

Especially in big companies or multi-project environments each department is addressed.

Brackets

Different departments, sourcesystems, projects, … leads to different field names over the time.

Example: The field containing „address“ has different spellings and it is hard to find all fields containing the same content over different sourcesystems.

With brackets you can group fields together with the same content to easy find them.

This can also be used to find fields containg GDPR relevant data in one click. Simple create a bracket which finds all fields with the patterns „surname“, „firstname“, „f_name“, „day of birth“, „birthday“.

Link everything to everything

Create line age where physically not present.

Do you have a URL to a intranet site which descriptes a process which is measures by a KPI? Link the meta#grid defined KPI to the URL. Otherwise is the URL automatically back linked to the KPI.

Does a field in the ERP act as source (e. g. SAP) for a dimension in the data warehouse (e. g. Snowflake database) which is transferred via ETL (e.g. Talend)? Those different systems do not have a physical connection or foreign key relation. Map those items together and see them in one simple list.

On the other hand are foreign keys automatically mapped if extracted with the bulk loader e. g. from a relational database. No manual work is needed if a physical relation already exists.