make printable

new: v1.0: - released

see also: webCMS syntax; intro presentation databases none [classification.txt will eventually go to a database] tqs tools webDocuSys; webNavbar output used by none upload/download upload (scp) created web pages to standard QC web server

intro presentation

webCMS

Description

The tool webCMS (web content management system) helps to create and maintain the QC web pages. The core ideas are:

• maintain the content on the operational machines and accounts,
• make it easier to focus on the content,
• provide a uniform design,
• make changes of the design easy while guaranteeing uniform look&feel,
• provide consistency and completeness checks during tool call,
• provide functionality beyond the CMS system, by linking the content to the webKPI and the webDocuSys systems.

The webCMS system maintains the HTML part of the tutorial web pages.

It does NOT maintain:

• any images,
• web pages outside the QC web pages branch.

Principles

As a content management system, webCMS has the structure of the pages (and the actual HTML code) encoded in the tool, while the content is stored in local text files and maintained by the user. It has a configuration file config.webCMS, a text file with the classification of the QC1 parameters (classification.txt), and text files with the content of the QC pages. The tool has its own directory $DFO_CONFIG_DIR/webCMS hosting all these files.

The file classification.txt will eventually be replaced by a database table. The tool also reads content from the file $DFO_CONFIG_DIR/webDocuSys/version_KPI.txt which will eventually be replaced by a database table. [KPIs are the QC1 parameters describing the instrument performance (literally: key performance indicators). There is a tool managing properties of the KPIs, webKPI (coming).]

The QC_SITE is the entire set of QC pages per instrument visible as the web site www.eso.org/qc/<instr>. It is defined and reflected in this structure.

The QC_SITE comes in three branches: HOME, QC_PAGE and PIPELINE. This structure is hard-coded in the tool. QC_PAGE is the set of tutorial pages. They appear under the navigation bar lable "Trending & QC". PIPELINE the set of pipeline and data documentation that appears under the lable "Pipeline". HOME has just one page, index_<instr>.html.

There is another branch, COMMON, which collects the content of the common pages which are embedded as general content in certain QC pages. These are maintained by a priviledged account which is hard-coded as 'giraffe@muc02'.

Each branch has its separate part of the URL. The QC_PAGE branch has 'qc', like www.eso.org/qc/GRAVITY/qc. Each branch consists of a set of web pages. The web pages for the 'qc' branch are collected under the navigation label "Trending&QC1". The web pages for the 'pipeline' part go to the label 'Pipeline'. The HOME branch gets a special, hard-coded treatment which can be ignored here.

A web page (PAGE) consists of one or several logical components. For the webCMS system, a component is the smallest logical unit. Its content is encoded as text file. All text files that constitute a web page are stored in the corresponding sub-directory PAGE, and are composed by webCMS into the web page. On the tutorial QC pages, a component typical refers to one HC page and can be recognized by its title.

The column VNAVBAR is filled with DEFAULT. Then the tool derives the name of the vnavbar from the PAGE name: for the PAGE bias.qc1.html, the vnavbar name becomes vnavbar_QC_bias_qc1.html if it is filled with DEFAULT. Only if the vnavbar has a different (non-standard) name, you fill this name.

As a special property, the pages of the QC_PAGE branch have some standard components (such as "Trending", History" etc.). Their existence is checked by the webCMS tool as a completeness and consistency check for the upcoming webDocuSys tool.

The way how the pages are organized (the sequence of web pages), and the internal structure of the web pages (the order of the components), is configured in config.webCMS.

The examples for the component files are the original source files for the corresponding webCMS pages

http://www.eso.org/observing/dfo/quality/GIRAFFE/qc/bias_qc1.html,

http://www.eso.org/observing/dfo/quality/GIRAFFE/qc/lamp_qc1.html,

http://www.eso.org/observing/dfo/quality/GIRAFFE/pipeline/pipe_calib.html.

Content

Most content in the text files is plain text, or related to interpreted high-level tags like TABLE or some ITEM identifiers.

TABLE content gets marked by table identifiers. ITEM identifiers are needed for some text files, there are just 2 or 3 and they appear at the begin of the file. A complete list of identifiers can be found here.

Figures get embedded into TABLEs, they have no type of their own. This makes sense because figures should always have a caption, and a figure with a caption naturally forms a table, or a row in a table. Other HTML elements like URLs or horizontal lines are embedded in TEXT and are interpreted as native HTML code. HTML tags should be kept to a minimum. Don't forget that the purpose of the webCMS tool is to eliminate the burden of HTML coding. In particular avoid anything like text decoration with colors. Likewise, font sizes have gone, they are replaced by classes and styles which are managed within the tool.

Syntax

The syntax and structural components are listed here.

Migration

1. Configuration

For a given page, you need to configure:

• its name in config.webCMS in the section 2.1 QUALIFIER, either in QC_PAGE or in PIPELINE (or in HOME);
• for the VNAVBAR, enter DEFAULT (unless you have a very good reason to use a different name);
• for the QC_TITLE, enter the title that you use on the current web pages; do not enter the instrument name, it will be provided by the tool.

Then, enter the COMPONENTs in section 2.2 .

• For the tutorial pages (QC_PAGE) start with 'top_table.txt', then add 'general.txt' (optional). The other pages (pipeline) don't have these files.
• Then add one text file per logical COMPONENT.

2. Create the directory $DFO_CONFIG_DIR/webCMS/${QC_SITE}/${PAGE}.

• Fill it with the configured txt files.
• Disable the config key INCLUDE_WEBDOCUSYS for the moment, this can be enabled later.
• From now on, you can start with the migration. IMPORTANT: make sure to call the tool like 'webCMS -p <page>', this will export everything to a URL containing 'qc_new' and 'pipe_new' instead of 'qc' and 'pipeline'. There you can do the comparion.
• If you call 'webCMS -p <page> -f' the tool will export to the real directories, then your new pages go life and overwrite the old ones. This should be the last step, obviously.

3. How to fill:

3.1 top_table.txt: use the provided example above, everything should be obvious.

3.2 general.txt: this is optional, and in many cases not needed. It is meant to provide some information before the structured part of the page. Here you can put information which is generally applicable to all components of that page.

3.3 Other components: at least one for QC_PAGE. For each component:

• open the current (to be migrated) web page in your browser; maybe it's also useful to open its local version in dreamweaver;
• copy&paste the content into the component txt file;
• remove all <font size=2> and </font> using 'sed'; no need for this anymore, the tool uses the style 'normal' everywhere which corresponds to font size=2;
• replace the heading HTML decoration by the identifier TITLE; this provides a standard look&feel and also automatically adds the navigation within the page;
  
  
• identify the content for the QC1_TABLE and for the HISTORY table, fill them in the way described in the TABLE_TYPES section;
  
• identify any other table with free content (not falling into these two catgeories), make your choice about the table type (likely: TABLE_CUSTOMHDR or TABLE_NOHEADER), and enter the content;
  
  
• for the figures, make your choice about FIGURE_1A/B/2
  
• at that occasion, please also check the URLs:
  • for anything including /qc or /quality: don't use the absolute ones (like http://www.eso.org/qc/...); instead use /qc/<instr>/etc. or even more relative ones (because of the replication to the 'pl' site);
  • only for sites NOT falling into the previous category (like 'http://archive.eso.org'), use their absolute URLs.
    
    
• drop all content above the title and after the last component: the header, the navbar, and the footer will be automatically added;
• there is no 'last update' anymore (since running the tool is not necessarily a content update);
• make sure to check anchor names (like #bias): these are defined as ITEM_ANAME and used for top-down navigation within the page; if you use them somewhere internally, make sure to check the old (hand-made) scheme versus this new automatic scheme.

4. Continue with the next pages until you are complete for the QC_PAGE branch.

• For simplicity you can use 'webCMS' without any parameter, then the tool loops over all existing pages.
• If you are sure that a certain page is complete, feel free to call 'webCMS -p <page> -f'.

5. Continue with the pages for the PIPELINE branch.

• They are less strictly controlled in terms of structure.
• Their tables and images are probably more demanding for migration.
• All remarks from step 4 apply here as well.

6. Finally add the index_<instr>.html page in the HOME branch, it has a general.txt file only.

Common files:

• these are the files that have common (not instrument specific) content; they are also managed by the tool but under priviledged access only; you need to make sure that they are embedded properly into your pages;
• use the tag VIRT_INCLUDE and the page name.

Simplifications:

• there is no TOC file (table-of-contents); please update your webNavbar configuration if not yet done;
• the files qc.html and qc1.html are entirely provided by common content, some other files are filled partially from there; these are the same files as before, they are hard-coded in the tool and you don't need to remember them; the tool will tell you if you have forgotten to include the tag VIRT_INCLUDE;
• there is no DATA_MANAGEMENT section anymore; make sure to remove it from your webNavbar configuration;
• make sure to change the webNavbar configuration for 'product_codes.html': it should go to 'pipeline' and not to the old 'ServiceMode'.

Content:

• the QC_PAGE branch is highly structured and needs to comply with webDocuSys; if you disable the key INCLUDE_WEBDOCUSYS (recommended for the initial migration), there will still be structural checks and the tool will complain about e.g. the section 'QC1 parameters' missing; follow the recommendations, add the sections and try again;
• the PIPELINE branch is not strongly structured, there are no structural checks;
• the name 'general.txt' is reserved for QC_PAGE content;
• make sure to remove remaining content mentioning 'data packages' (this is really long ago now and does not need to be kept);
• products, master calibrations etc.: refer to things which are visible to an archive user (calSelector); there is no need for mentioning 'pipeline products not archived', of QC reports etc.

Checking:

• the tool checks for the existence of 5 standard TITLEs for QC_PAGE components:
  • QC1_parameters
  • Trending
  • Scoring&thresholds
  • History
  • Algorithm
• they have to exist as lines starting with the proper TITLE; if missing, the tool raises a warning but continues;
• nevertheless it is a good idea to provide these TITLEs already now since they will be checked more strictly with webDocuSys.

File classification.txt

The text file classification.txt under $DFO_CONFIG_DIR/webCMS is important. It is used by two tools: webCMS and webDocuSys. It will eventually go into a TBD database and then also have an interface to create/delete/modify content. Currently it is a simple text file structured like a config file.

It contains one row per QC1 parameter and HC plot. All QC1 parameters that are used in a HC plot should be listed here. This is the core idea of the upcoming webDocuSys tool. The documentation information for this QC1 parameter should then also be made visible on the tutorial pages (the QC_PAGE branch of webCMS), which are also linked to the HC plots.

If a QC1 parameter has entries in a QC1 database table but is not used in any HC plot, don't list it here. If you create a new HC plot, always add the corresponding records in classification.txt. If you remove a HC plot, remove all entries referring to that plot.

The table has some content which is not evaluated by the webCMS tool. It should nonetheless be filled consistently.

In the contect of this table, a HC plot is any of:

• a genuine HC plot having the tag _HC (standard case);
• a KPI plot having the tag _KPI (coming, few cases);
• a FULL plot that has no HC plot (rare).

This boils down to the rule that every config.tp file should have at least one entry in classification.txt, with the exception of:

• MONitoring plot gets ignored (RANGE column);
• FULL plots existing in addition to HC plots get ignored.

Often there will be more than entry, because all plotted QC1 parameters per HC plot should have one entry.

Content:

Find the operational classification.txt for GIRAFFE here. Each record has the following columns:

HC_CLASS:

There are four classes. A QC1 parameter can have more than one class although this rare. Think of the classes as a role. For instance, the read noise of the standard read mode is likely a KPI while the same parameter, for a rarely used read mode, is a class HC or CAL.

A parameter used in a KPI plot has always class KPI. All other parameters describing instrument properties important for data quality are classified HC. Parameters that describe the data quality but are not instrument properties (like count level in a flat) are classified CAL. Parameters which are not related to quality but are needed e.g. for correlation (like temperature) are classified ENG (for engineering).

The item QC1_parameter needs to be filled by table..parameter in order to be unique per instrument.

The items TPL_ID and RECIPE are needed for webDocuSys.

How to use

Type

webNavBar -h for on-line help,

webNavBar -v for the version number.

The standard mode is:

webCMS -p <page_name>

where page_name is entered without .html, e.g.:

webCMS -p bias_qc1

The tool reads the components as configured in config.webCMS under COMPONENT (section 3), with the sorting order preserved. The content is interpreted (TABLE names, IDENTIFIERs, QUALIFIERs etc.). If configured, a COMMON part is added. The page <page_name>.html is created, the vnavbar, the top and bottom parts and the title(s) are added.

Finally the page is transferred to the URL www.eso.org/qc/<instr/qc_new/<page_name>.html (or pipe_new for the PIPELINE branch or home_new for the HOME branch). The page is then available for comparison to the operational page which is under www.eso.org/qc/<instr/qc/<page_name>.html (or pipeline for the PIPELINE branch).

This mode is also called the development mode because you can compare the original and the new content, without overwriting the original content.

Once you have checked for completeness and for correctness, call

webCMS -p <page_name> -f

which will transfer the page to the operational URL. This is the 'forced' mode using the -f flag (as trendPlotter does), and the page will go life.

Now the migration for <page_name> is finished. Future maintenance will be done on the local text files in your operational muc environment. The original page (before migration) will probably still exist on your local PC but it will not be used anymore.

Calling the tool in global mode without parameter:

webCMS

This will create ALL configured and existing pages in a loop, in development mode. The tool reads the config file to get the branches and all page names and then loops over all page names.

Calling

webCMS -f

will finally refresh the entire website.

Creating a directory for a new page, and filling it with txt files, will have no impact if you forget to configure the new page in config.webCMS. Think of this as a "registration": an unregistered page does not exist for the tool.

In a later stage, and upon further trigger, the tool can be 'upgraded' to include information from webDocuSys in the QC tutorial pages. Check the GIRAFFE QC pages for a prototype implementation. This is currently not supported for any other instrument, so set INCLUDE_WEBDOCUSYS to NO.

Output

Configuration

The tool has a tool configuration file, the file classification.txt (see above), and a folder tree HOME, QC_PAGE and PIPELINE. All these go to $DFO_CONFIG_DIR/webCMS.

The tool configuration file (config.webCMS; find the operational example for GIRAFFE here):