Changes between Initial version and Version 1 of building_the_docs

28 Dec 2017, 11:31:04 (4 years ago)



  • building_the_docs

    v1 v1  
     3Here put examples on how to build the docs
     5Logs of documentation writing:
     7* Building the docs:
     15* generating overview:
     17* Fixing issues:
     19* and locales (1/2):
     21* locales (2/2)
     22* next time
     23* sphinx commands / transifex
     26== Required softwares ==
     28* CMake (>= 2.8)
     29* git
     30* perl:
     31** 1 hour tutorial:
     32* sphinx (>=1.1)
     34=== Installing dependencies ===
     36On <code>ubuntu 16.04</code>
     38<source lang="bash">sudo apt install cmake git
     40sudo pip install sphinx</source>
     41== Preparing the repository ==
     43=== cloning the repository ===
     45=== branching ===
     47== Build commands ==
     49=== Simplest build ===
     51This build will only make the English version and will be faster so you can quickly visualize your changes.
     53<source lang="bash">## making the build folder (where builds will be put)
     54mkdir build
     56## Getting into the build directory
     57cd build
     59# Preparing the build
     60cmake -DHTML=ON ..       # don't forget the .. at the end !!
     62# Build html files
     63make html
     65And you're done ! You can go to <code>build/doc/_build/</code> and open <code>index.html</code> in a web browser.
     67=== Main workflow ===
     69* make the build directory <code>mkdir build</code>
     70* get into build/: <code>cd build/</code>
     72==== Prepare the build with CMake ====
     74With <code>cmake -D&lt;build option&gt;=ON  ..</code> with the option in uppercase:
     76<code>cmake -DSINGLEHTML=ON  ..</code>
     78==== Available options ====
     80Available options can be see with the command:
     82<code>cmake -L ..</code>
     84===== Building options =====
     86There is several option available to the build:
     88* HTML (plain HTML)
     89* DIRHTML (HTML pages with internal links not working)
     90* SINGLEHTML (All the docs in a single html page)
     91* PICKLE
     92* JSON
     93* HTMLHELP
     94* LATEX (LaTeX format, used to generate pdfs)
     96===== Languages options =====
     98As for now the supported languages are:
     100* &quot;ca&quot;
     101* &quot;de&quot;
     102* &quot;es&quot;
     103* &quot;fr&quot;
     104* &quot;it&quot;
     105* &quot;ja&quot;
     106* &quot;ko&quot;
     107* &quot;pl&quot;
     108* &quot;ru&quot;
     109* &quot;hu&quot;, &quot;el&quot;, &quot;id&quot; and &quot;zh&quot; are not available yet (bug fixe in progress)
     111By default, english will be always made. You can add languages easily with the flag <code>-D&lt;Language code&gt;=ON</code>
     113For example, to get spanish:
     115<code>cmake -DHTML=ON -DES=ON</code>
     117To use a specific language, use the cmake flag <code>-D&lt;language code in uppercase&gt;=ON -D&lt;another language code in uppercase&gt;=ON</code>
     119You can pass no language flag (building only english) or as many flag as there is supported languages (and built all of them).
     121To build all languages you can use the flag <code>-DALL_LANG=ON</code>
     123The option (<code>-DES=ON ...</code> or <code>-DALL_LANG=ON</code>) are kept in the <code>OSGeoLiveDoc_BUILD_LANGUAGES</code> variable, which is automatically filled by CMake.
     125==== make the build ====
     127<code>make &lt;option&gt;</code> with option in lowercase:
     129<code>make singlehtml</code>
     131==== about Sphinx ====
     133Sphinx generates the po files used by transifex.
     135Sphinx needs locales, english is set by default. Other locales are generated from the languages in <code>OSGeoLiveDoc_SUPPORTED_LANGUAGES</code> variable in <code>CMakeLists.txt</code>. For more details, see above.
     137== Projects management ==
     139Projects are loaded from the <code>projects_info.csv</code> file. It is the main entry point.
     141<blockquote>Note: Actually, it is manually build from this file:
     143To add a project or update project version, modify the projects_csv file:<PATH TO FILE>/projects_info.csv
     145Please indicate if there is a quickstart or not, an overview or not, an openHub account or not, the version, if it is an OSGeo project, or community or none of those.
     147Overviews will be generated with the data from <code>projects_info.csv</code>, and links to quickstarts will be added automaticly.
     149More information how CMake use this file are available above.
     151== Debugging ==
     153There is 2 versions: regular and verbose.
     155The first one can be called with this flag: <code>-DOSGeoLiveDoc_DEBUG=ON</code>
     157For example: <code>cmake -DHTML=ON -DES=ON -DOSGeoLiveDoc_DEBUG=ON ..</code>
     159The verbose debug is called with this flag: <code>-DOSGeoLiveDoc_VERBOSE_DEBUG=ON</code>
     161For example: <code>cmake -DHTML=ON -DES=ON -DOSGeoLiveDoc_VERBOSE_DEBUG=ON ..</code>
     163== Configuring CMake / Setting the build ==
     165CMake is use to build the repository instead of simply make because it can be finely tuned. CMake configuration is done by the <code>CMakeLists.txt</code>. CMake says how the build is made from that file.
     167This file allows to set a few things:
     169==== Checking CMake version ====
     171<code>cmake_minimum_required(VERSION 2.8 FATAL_ERROR)</code>
     173==== Checking build folder ====
     175The first thing that CMake does is to check if you are in the folder called <code>build</code>. If it is not the case, CMake will return an error.
     177<pre class="{bash}">if ( ${CMAKE_SOURCE_DIR} STREQUAL ${CMAKE_BINARY_DIR} )
     178message(FATAL_ERROR &quot;In-source builds not allowed.
     179Please make a new directory (called a build directory) and run CMake from there.
     180You may need to remove CMakeCache.txt.&quot; )
     182==== Changing the version number ====
     184In <code>CMakeLists.txt</code>, you can change the OSGeo-Live version number with the following variables:
     186<pre class="{bash}">## Major version
     187set(OSGeoLiveDoc_VERSION_MAJOR &quot;12&quot;)
     189## minor version
     190set(OSGeoLiveDoc_VERSION_MINOR &quot;0&quot;)
     192## Patch number
     193set(OSGeoLiveDoc_VERSION_PATCH &quot;0&quot;)
     195## if it is a development version, a Release Candidate or a final release
     196set(OSGeoLiveDoc_VERSION_DEV &quot;-dev&quot;)</pre>
     197==== Checking the sphinx version ====
     199To build, you sphinx 1.1 and above:
     201<code>set(SPHINX_MINIMUM_VERSION &quot;1.1&quot;)</code>
     203If not a warning will be displayed. If you need a more recent version to build the doc, you can test it from here.
     205==== Checking Perl ====
     207Perl is needed at some point, so
     209==== Adding / removing a language ====
     211Add the 2 letters code of the new language:
     213<source lang="bash">set(OSGeoLiveDoc_SUPPORTED_LANGUAGES "ca"  "de"  "es"  "fr"  "it"  "ja"  "ko"  "pl"  "ru")</source>
     214Then add a new <code>INSERT_INTO_MAP</code> command to add the language to the lang navigation bar (in <code>index.html</code>). For example:
     216<code>INSERT_INTO_MAP(&quot;fr&quot; &quot;Français&quot;)</code> in the <code>CMakeLists.txt</code>.
     218The line <code>set(OSGeoLiveDoc_ENGLISH &quot;en&quot;)</code> is not supposed to change, it sets the default build in english.
     220=== Projects management ===
     222Projects are loaded from the <code>projects_info.csv</code> file. It is the main entry point.
     224<blockquote>Note: Actually, it is manually build from this file:
     226To add a project or update project version, modify the projects_csv file:<PATH TO FILE>/projects_info.csv
     228Please indicate there is a quickstart or not, an overview or not, an openHub account or not, the version, if it is an OSGeo project, or community or none of those.
     230Overviews will be generated with the data from <code>projects_info.csv</code>, and links to quickstarts will be added automaticly.
     232== reading the file ==
     234<code>file(STRINGS projects_info.csv OSGeoLiveDoc_PROJECTS_VERSIONS_FILE)</code> opens the csv file
     236<code>set (OSGeoLiveDoc_PROJECTS_VERSIONS &quot;&quot;)</code> is prepared to stock data from the file
     238Data are extracted with a regular expression, which detect the column, but has to know the number of columns. Currently the file has 9 columns.
     240The file is processed line by line.
     242== Slicing the lines with a regex ==
     244<code>set (OSGeoLiveDoc_CONFIGURATION_REGEXP &quot;^(.*)\\|(.*)\\|(.*)\\|(.*)\\|(.*)\\|(.*)\\|(.*)\\|(.*)\\|(.*)&quot;)</code>
     246So <code>(.*)\\|</code> has to be repeated 9 times. If a column is added in projects_info.csv, it is mandatory to add one more <code>(.*)\\|</code> in the regex to find the new column.
     248Each column will extracted by group, for example, column 3 will be extracted as group &quot;\3&quot;; because it's a csv, it's kind of a 1 to 1 relationship between the group and the column.
     250Quick explanation of the regex:
     252^ &lt;--- beginning of line (.'') &lt;---- any character = &quot;.&quot; any number of times = &quot;''&quot; grouped &quot;(&quot; &quot;)&quot; \| &lt;--- code for vertical line &quot;|&quot;
     254The for each line in the file, the ones starting with '''#''' are treated as comments. Comments will be ignored and the whole line replaced by a <code>#</code>.
     256This part only use the 5 first column.
     258== Group use ==
     260For example, group &quot;\3&quot;, will get the project version, and will be cleaned with <code>STRIP</code>:
     262<code>string(STRIP ${project_version} project_version)</code> will strip the blanks spaces at the beginning and the end of the string. So &quot; 4.2.1 &quot; will return &quot;4.2.1&quot; and &quot;4.2.2 beta &quot; -&gt; &quot;4.2.2 beta&quot;.
     264=== Debugging ===
     266This treatment can be seen with the <code>-DOSGeoLiveDoc_VERBOSE_DEBUG=ON</code> flag. It will show something like that:
     268-- Y 52nSOS 4.4.0 want_quickstart:Y want_overview:Y -- Y 52nWPS 3.6.1 want_quickstart:Y want_overview:Y -- N 52nWSS retired want_quickstart:N want_overview:N -- 52nWSS is not for documentation -- N atlasstyler retired want_quickstart:N want_overview:N -- atlasstyler is not for documentation -- N cartaro retired want_quickstart:N want_overview:N -- cartaro is not for documentation
     270Active projects will get want_quickstart and want_overview set to Y; the retired projects will be set to N and won't be used for documentation.
     272This process is always done, but won't be print unless the <code>-DOSGeoLiveDoc_VERBOSE_DEBUG=ON</code> flag is used.
     274=== Project version in ===
     276For each line (so active and retired projects), a new python command will be generated for inserting in
     278<code>.. |version-${project_name}| replace:: ${project_version}\n&quot;</code>
     280For example, for the ''4.4.0'' release of the ''52nSOS'' project, the string will be <code>.. |version-52nSOS| replace:: 4.4.0</code>.
     282=== Selecting project who needs on overview and a quickstart building ===
     284This code add the project name to lists for quickstart and overviews, if a Y is the columns/ groups 4 and 5. Some projects might have an overview but no quickstart, it is mostly libraries.
     286if (&quot;<math>{want_quickstart}" MATCHES "Y") list(APPEND OSGeoLiveDoc_QUICKSTART "</math>{project_name}&quot;) endif() if (&quot;<math>{want_overview}" MATCHES "Y") list(APPEND OSGeoLiveDoc_OVERVIEW "</math>{project_name}&quot;) endif()
     288= Standards =
     290Standards are builded with this set of commandes:
     292set(OSGeoLiveDoc_STANDARDS &quot;csw&quot; &quot;gml&quot; &quot;sld&quot; &quot;wcs&quot; &quot;wms&quot; &quot;fe&quot; &quot;kml&quot; &quot;sensorml&quot; &quot;sos&quot; &quot;wfs&quot; &quot;wps&quot; ) # Other settings There is several files that is need for the build, they are copied with the command <code>configure_file()</code>
     295* licences.csv
     296* contributors.csv
     297* translators.csv
     299For example, <code>configure_file(&quot;; &quot;;)</code> will copy '''' and '''substitute''' the variables defined in the cmake. Which is very handy.
     301The csv files are copied but are not modified. They'll be loaded later in the html file with an another process.
     303= Setting building variables =
     305The line <code>add_subdirectory(doc)</code> will add the '''doc''' folder to the build and all the files in it will be set with the variables from cmake, by copying the CMakeLists.txt file in each folder and subfolder. Each copy
     307= Comments in CMake =
     309There is no comments in CMake, but there is a way to have code that won't be executed.
     311if(FALSE) <none executed code> endif()
     313= Included files and folder =
     315CMake will use the files and folders in the main folder (aka <code>OSGeoLive-doc</code>) but only the ones present in the variables <code>OSGeoLiveDoc_FILES</code> and <code>OSGeoLiveDoc_FOLDERS</code>.
     317In CMakeLists.txt they are set like that:
     319<pre class="{bash}">#---------------------
     320# Files
     343# Directories
     353So commented out or deleted lines won't be taken in account, and won't be translated. If you comment &quot;overview&quot;, the overviews won't be built.
     355= Directory processing =
     357This process is managed by the command <code>add_subdirectory()</code> which tell '''CMake''' to look for a ''CMakeLists.txt'' in the subdirectory and execute it.
     359For example, the last line of the main ''CMakeLists.txt'' is <code>add_subdirectory(doc)</code> so '''CMake''' will look into doc/ and execute the ''CMakeLists.txt'' in it. Several variables will be initialized like <code>${OSGeoLiveDoc_SUBDIRS}</code> which contains the folders to look into.
     361Then the following loop will be looking in each named subfolder to execute its ''CMakeLists.txt''
     363<code>{bash} foreach (dir ${OSGeoLiveDoc_SUBDIRS}) message(STATUS &quot;   Adding directory ${dir}&quot;) add_subdirectory(${dir}) endforeach()</code>
     365For example, the proccessed files for the Overview folder are extract from <code>${OSGeoLiveDoc_OVERVIEW})</code> built in the previous process (from <code>projects_info.csv</code>). Files not listed in <code>${OSGeoLiveDoc_OVERVIEW})</code> won't be processed.
     367== Local list ==
     369Some ''CMakeLists.txt'' have a local list, for example in quickstart, there is local list for non project quickstart. It is quickstarts non included in the ''projects_info.csv''.
     371If a new project add its ''.rst'' file in the quickstart folder but forget to update ''projects_info.csv''; the quickstart won't show.
     373For a non project quickstart like ''virtualization_quickstart'', which explain how to use OSGeo-Live in a virtual machine, it has to be set in the '''LOCAL''' variable of the quickstarts ''CMakeLists.txt''.
     375== Quickstart and Overview folder ==
     377The CMakeLists.txt here to copy all projects listed in <code>projetc_info.csv</code> and transmitted with the variables <code>${OSGeoLiveDoc_QUICKSTART}</code> and <code>${OSGeoLiveDoc_OVERVIEW}</code>.
     379Local CMakeLists.txt will be initialized with those variables.
     381=== Overview folder ===
     383in Overview, there is no local files. The Overview webpage (listing all projects) is generated with perl from data extracted of <code>projects_info.csv</code>.
     385== Standards ==
     387Standards were not part of the documentation any more, they have been removed. The process is still there, in case they come back into the documentation.
     389== Main files ==
     391Main files, present at the root of the project and website, are processed if listed in the variable <code>OSGeoLiveDoc_FILES</code> at the beginning of the ''CMakeLists.txt''. They can be added and commented out easily.
     393= Images =
     395The images folder is entirely copied to <code>build/doc/images</code>. All images are copied, even the one that are not used by the documentation any more. That need to be cleaned.
     397= Scripts =
     399Some files are build with perl scripts like <code>overview.html</code> or <code>metrics</code>. Those scripts will be detailed below.
     401== overview.html ==
     403[ overview.html] is the overview of all projects, and is generated from data in <code>projects_info.csv</code>. So retired project wouldn't appear in it.
     405The script is there:
     407First three lines are testing the shell and args.
     409There is includes (<code>pragmas</code>) from perl library used by the script in line 6 to 8.
     411Then three specials variables are declared: - ''name (filename) - ''dir (get the directory of the perl script) - *prune (get the path to the perl script)
     413They are shortcuts for functions (<code>*File::Find::name</code> is a function returning the filename).
     415Those varaibles are loaded with informations from the CMake: * <code>$DEBUG = &quot;@OSGeoLiveDoc_DEBUG@&quot;;</code> * <code>$version = &quot;@OSGeoLiveDoc_VERSION@&quot;;</code> * <code>$projects_info_file = '@CMAKE_SOURCE_DIR@/projects_info.csv';</code> * <code>$output_file = '@CMAKE_BINARY_DIR@/doc/overview/overview.rst';</code>
     417==== See if the <code>projects_info.csv</code> file exist, if don't the script stop and throws an error: ====
     419<code>die &quot;ERROR: Failed to find: '$projects_info_file'\n&quot; unless -f $projects_info_file;</code>
     421=== Get the configuration from projects_info.csv ===
     423This line parse the projects_info file and stores it in <code>$configuration</code>. If the flag <code>-DOSGeoLiveDoc_DEBUG=ON</code> is used, then a comment line will be print.
     425<source lang="perl">my $configuration = read_and_parse_configuration($projects_info_file);</source>
     426==== read_and_parse_configuration subroutine ====
     428This subroutine takes a file in parameter. It is stored in $file with the keyword <code>shift</code>.
     430Then an empty Hash called %hash is initialized. It will store hashes extracted during the process.
     432The script will try to open the file in parameter or die and throws an error.
     434Then a while loop reads the file line by line and exit when the file is empty (operation becomes <code>TRUE</code>). During the while loop several actions are made: - ignores the commented lines; - clean the line by removing spaces surplus; - ignores project who are not subject to documentation (no overview and / or no quickstart) - get lines in the right bucket (section) <code>$hash{$values[5]}</code>
     436For example, with the following line:
     438<code>Y | 52nSOS | 4.4.0 | Y | Y | Web Services | | Sensor Observation Service | SensorObservationService</code>
     440The instructions will put the line in the bucket that has the name ''Web Services''.
     442At the end of the loop, the file is closed and the hash with the collected data is returned.
     444In the script, the hash will be store in <code>$configuration</code>.
     446=== Building sections ===
     448Sections are stored in <code>$sections</code>.
     450For each defined sections, there will be a call for the section in <code>$configuration</code>. It use the subroutine <code>get_section</code>.
     452The result will be append to <code>$sections</code>.
     454==== get_sections() ====
     456This subroutine take two parameters: * the name of the section * the variable where the configuration is stored
     458<blockquote>Note: There is a lot of concatenation in this subroutine. It is short written with <code>.=</code>
     460get_sections() workflow: - prepare the section headers and toc tree - split values around the <code>|</code>and store it in an array (@values) - cleans the excess <code>|</code> and spaces in in each value - Test it ''overview'' is set to <code>Y</code> then handle the overview by adding it to the toctree and adding the bullet line - Test it ''quickstart'' is set to <code>Y</code> then handle the quickstart by adding it to the toctree and adding the bullet line between brackets - Write the comment / short description if provided.
     462When all projects have been read, the toctree is added to the contents, then the bullets. The contents is returned.
     464=== Writing the file ===
     466When all the sections are build and added to <code>$sections</code>, the subroutine <code>write_script()</code> is called.
     468It takes one parameter, <code>$sections</code> with all the sections, in the right order.
     470# it tries to create the output file (or die if cannot)
     471# write out the header and the commands to clean up the old extension
     472# paste the content of <code>$sections</code> after the toc (<code>.. contents:\n :local:</code>) and before prior applications
     473# close the file
     475== Script exit ==
     477When the script is done, it exit returning <code>0</code>. It is a mandatory for perl.
     479= =
     481=== Variables ===
     483The variables are loaded by CMake during the build from data in <code>CMakeLists.txt</code>.
     485<source lang="perl">my $DEBUG = "@OSGeoLiveDoc_DEBUG@";
     486my $version = "@OSGeoLiveDoc_VERSION@";
     487my $projects_info_file = '@CMAKE_SOURCE_DIR@/projects_info.csv';
     488my $output_file = '@CMAKE_CURRENT_BINARY_DIR@/../metrics.rst';</source>
     489= Useful commands =
     491=== Rename or move files and rename all occurences ===
     493==== Moving the files to the right folder ====
     495<source lang="bash">git mv doc/images/projects/gvsigFOOOOOOOOOOOO doc/images/projects/gvsig
     497# see the change
     498git status</source>
     499==== Changing all occurrences ====
     501To do that, we will use a one liner perl command to change from projects/gvsigFOOOOOOOOOOOO to projects/gvsig
     503the <code>/</code> is special in perl so, <code>\</code> is prepended in the command from projects/gvsigFOOOOOOOOOOOO to projects/gvsig
     505==== THE command: ====
     507<source lang="bash">perl -pi -e "s/projects\/gvsigFOOOOOOOOOOOO/projects\/gvsig/" $(git ls-files)</source>
     508* <code>git ls-files</code> : lists the files that git knows about
     509* <code>perl</code> &lt;--- use that
     510* <code>-pi</code> &lt;--- one liner
     511* <code>-e</code> &lt;-- do changes on the file and save them on the file
     512* &quot;s/projects/gvsigFOOOOOOOOOOOO/projects/gvsig/&quot; &lt;&lt;&lt; s/<from>/<to>/ substitute from to
     513* <code>$(git ls-files)</code> &lt;---- $(<command>)
     514* <code>git ls-files</code> &lt;--- generates filenames know to git
     516= Check and remove unused images =
     518When updating the docs, it can happen that some images are not removed. So the [] script check all the projects (current and retired).
     520= =
     522Sphinx need a file with several data. With this project, it is built from [] where @var@ are modified with data from CMakeLists.txt.
     524The original [] should be remove at the end of the project. This is done a the [ L. 76 of CMakeLists.txt]
     526= Theming =
     528Themes are stock in the <code>_theme</code> subdirectory, they are loaded by the line [ L. 79 of CMakeLists.txt]
     530There is a small [ local CMakeLists.txt] for each theme. It is used to configure the language bar with the asked languages.
     532= Locales =
     534Locales are stocked in the (locales folder]( This folder is updated by 2 distinct processes:
     536* first time, by sphinx, when building the strings to be translated (before pushing it to '''Transifex''')
     537* second time, when the translated strings are retrieved from [ Transifex]
     539Waiting for [ issue #44] to be fixed.
     541= Sphinx commands =
     543The [ lines l.115 to l.156] are sphinx commands. Those commands are used to communicate with the [ Transifex plateform].