Open

For GDMS, the following error validation checks on upload are critical and if they do not pass will block the data at upload form even being validation:

• The uploaded file must be less than 1GB.

• The uploaded file must have the filename extension ".zip" and be a ZIP file.

• The uploaded ZIP file must be readable and non-encrypted.

• The uploaded ZIP file must not contain any viruses or other malware.

• The ZIP file must only contain files of the following types: .tif, .tiff, .csv, .png, .jpg, .txt, .pdf, .jpeg, .doc, .docx, .dxf, .emf, .dwg, .dwf, .xls, .dxf, .dgn, .xlsx, .xlsm, .dbf, .shx, .shp

• The ZIP file must not contain any subfolders.

• The uploaded ZIP file must contain the files: point.shp, point.shx, point.dbf

• If the uploaded ZIP file contains a continuous.shp file, it must also contain the files: continuous.shx, continuous.dbf and component.dbf

• If the uploaded ZIP file contains a region.shp file, it must also contain the files: region.shx and region.dbf

• The uploaded SHP files must not contain a "SUPP_SCH" field.

• The uploaded SHP and DBF files must as a minimum include ASSET_REF and SUPP_REF fields.

• The value of fields where populated must be appropriate to the field type.

There are a number of things to consider that all affect the results that are output from WinCan VX during the export process, and they are almost all handled automatically in the background during the data exchange routine without any user options available to the user.

Theses are driven by:

• The difference in core data structures between GDMS and WinCan where GDMS stores all of its asset and inspection data in a single file (i.e. continuous, point and region etc) but WinCan VX (unlike WinCan v7) stores these very different types of data in different tables in its database.

• The round tripping process where any assets imported into WinCan Vx at the start of the round tripping process that are not inspected are simply returned back out of the database and back into GDMS via the shapefiles without any modification.

So, if we just import some GDMS shapefiles and then export them back out again, there will be no changes in the data.

If we import 1,000 sections in a catchment and then only inspect 40 of them on a job, then we will export the new data for the 40 inspected objects and the remaining 960 sections will just be exported back out again without edit.

This is why it is crucial that sections laterals and nodes that were imported at the start of the process are not deleted from the WinCan VX project, and also why good use of the ‘Inspection Status’ field is crucial to managing the data efficiently.

In the GDMS data files, there is only ever one combined inspection and asset data record of each object, but of course, WinCan VX has the concept of ‘multiple inspections’ and ‘combined inspections’, so how do we decide which inspection to export when there are several to chose from? This is where the Export Hierarchy comes into play, and the selected export order of the one inspection record to be exported (applies to all object types) is like this:

1. Combined inspection - if a combined inspection record exists on an object then it is exported and no other inspection data is exported for the current asset.

2. A complete inspection from end to end - this might not be the most recent inspection because maybe there was am abandoned inspection carried out after the complete one, but this does not matter under GDMS rules - a complete inspection trumps an abandoned inspection regardless of date and time.

3. An abandoned inspection - the most recent abandoned inspection will be exported. Of course if there was at least one abandoned inspection from each end then there should be a combined inspection, so we are back up this tree to option 1.

4. The original imported data, untouched, unvalidated, not re-scored and unmodified.

At the end of this export process, every asset has an inspection record exported into the data regardless of whether it was inspected by the contractor during this package of work or not.

The files that are produced by the GDMS export routine are described in the next section, but are essentially only the files required by GDMS for upload and a .dxf file. The important part about this is that any other deliverables that your client may like to receive have to be created separately from the export routine.

WinCan VX has all of the tools and features to do this without the need for other software applications, but these commonly requested files are not part of the clearly defined upload data for the GDMS system. They are additional requests made by the area clients, and the contractor should be pricing to deliver these over and above the standard deliverables.

Before running the export routine, the user is strongly advised to revisit and evaluate the following checklist in the order given and ensure that all items are 100% satisfied, because failure to do this will almost certainly result in critical rejection errors when the shapefiles are attempted to be uploaded to GDMS:

1. Run the ‘Duplicates Finder’ tool on Sections and Nodes from the Edit ribbon (Tools → Misc Tools list before VX v15.2) and resolve issues.

2. Execute the ‘Delete Empty Inspections’ tool from the Edit ribbon (Tools → Misc Tools list before VX v15.2) list.

3. Execute the ‘Regenerate Sort order’ tool from the Edit ribbon (Tools → Misc Tools list before VX v15.2) list.

4. Run the inspection merging tool using the default options as described here.

5. Run the ‘Calculate Invert Levels’ tool on the project.

6. Run the ‘Job Checker’ tool from the Edit ribbon (Tools → Misc Tools list before VX v15.2) list and resolve issues.

7. Validate the Sections and clear all Errors to zero.

8. Validate the Laterals and clear all Errors to zero.

9. Validate the Nodes and clear all Errors to zero.

The first step in the export process is to click on the ‘Export’ button in the ‘Data Exchange’ ribbon:

Open

The options selected will be correct for the current project, so there is nothing more to do here except click the ‘Next’ button

At the next screen, there is also nothing to change except if you wish to export the data only without the media files. The default settings are fine as they are.

Click the ‘Next’ button for a final review of what is going to happen during the export process, and then ‘Next’ one more time to execute the export routine.

When the process has finished, you maybe presented with some warnings, but provided the final message says that the ‘Process has been finished’, then you are good to go, and clicking on the folder icon to the right will take you straight to the export file folder ([Project Folder] → Misc → Exchange → [Project Name]_Date_Time and the folder containing the export files will be named [Project Name]_GDMS_2024. Inside here, you will find the export files as described:

Open

A common question after export is ‘Why has my data now got Auto ghost nodes in it after export?’

There are 2 drivers of auto created nodes inside the data after the export routine has been executed. The first is clearly visible in the WinCan project after export, and the second is only visible in the shapefiles with no change in the project, although the trigger for the change will be clear.

After export of the data to the GDMS shapefiles, you may see some additional nodes in your WinCan project named Auto_GN_nnnn where nnnn is a numeric counter.

These have been created automatically by the export routine to satisfy one of the ‘golden rules' of GDMS data for a successful upload. This rule is that every continuous item must have a point item at each end that is in the same catchment scheme.

Imagine this scenario:

Open

It is not possible to enter data into GDMS like this, because Pipe 2 has one node in catchment 1 and one node in catchment 2.

So, if you have a section that is in catchment 1 and it has a node at each end where one node is also in catchment 1, but the other node is in catchment 2, then a Auto_GN node will be created in catchment 1 and linked to the section to satisfy this golden rule. The new Auto node will be a Ghost Node at the same coordinates as the original node, so now you have 2 nodes on top of each other in the same place, like this:

Open

The drawing has been exploded to demonstrate what has happened here, but you should read this so that that Manhole 2 and Auto Ghost Node 1 are at exactly the same position - they are on top of each other.

Now the data is good to go to GDMS, because Pipe 1 has a node at each end that is in the same catchment 1 as the continuous item, and Pipe 2 also has a node at each end that is in the same catchment 2 as the continuous item.

This feature is somewhat complicated by the fact that only inspections are linked to catchments, not assets. So, on investigation, you might have:

• A section with multiple inspections - you must check that all of the inspections of the section are in the correct catchment and make amendments if necessary.

• Nodes connected to the section which might also have multiple inspections - you must check that all of the inspections of the nodes connected to the section are in the same catchment as the section.

There is a tool available in WinCan VX 10.0 and higher which helps to resolve these issues before export:

Open

When launched:

• The application checks that all section inspections are in the same catchment as the original imported catchment from the very beginning of the process.

  • If they are not, then it automatically assigns the original read-only catchment to all inspections of the section.

  • It then goes on to do the same thing for all the inspections of the 2 nodes that are attached to the section, so the catchment information is auto-fixed in this scenario.

• If the section is a newly added section that did not exist in the original data, then it checks that all inspections of the section are in the same catchment.

  • If they are not, then the user is prompted to select a catchment from the available list for the section.

  • Then, the inspections of the nodes connected to the section are moved to this same catchment also.

A common user error when checking the cohesion of section catchment IDs against its node catchment IDs is to not consider multiple inspections - it is crucially important to align the catchment ID of all inspections of the section first with the original catchment ID of the read-only inspection (if it exists), or with a user selected catchment if it is new data, then align all of node inspections of both nodes to the same catchment ID as the section.

Due to the addition of Components in the GDMS dataset over DDMS, there will no longer be any Auto_SC/DC/LC/MC_nnnn nodes created in the shapefiles during the export process. However, import data can easily contain them from previous DDMS exports.

Where these exist, care should be taken to re-join these split assets after importing the original shape files, before any work begins. Please see here for more information.

After running a GDMS export, there will usually be changes that have been made to your project during the export routine. This has been mentioned already here. There will also be a backup made of you’re database files immediately before the export was run, so…

If you are unhappy with the database after export, you must immediately roll back the database files, investigate the issues and try again.

The most common issue with bad data after export is Auto_GN_nnnn nodes as described above. These auto nodes are created with good intention to help you deliver GDMS compatible data, but it should only happen very rarely with ‘good’ data.

So, after export, check out your node list and if there are a lot of Auto_GN_nnnn nodes, then do not go any further. Roll back the database, investigate and fix the issues. The issue will most likely be linked to the misalignment of Catchment IDs between the continuous item inspections and their two linked point item inspections.

How many is ‘a lot’ ? There is no fixed answer to this, but in a perfect world, there will be zero auto ghost nodes created during export. As an example, if you imported 2 GDMS catchments into VX and then just exported them again, there will be no auto ghost nodes created, because there will have been no misalignment of the continuous-point relationship in the imported data.

But, when working on site, we modify and update the data, and we add new assets, so this is where genuine discrepancies may creep in to the project.

As a rule of thumb, the number of auto ghost nodes created should be in single figures, and very close to zero. Regardless of how many auto nodes are created, they should all be investigated closely to see if they really are genuine or if they have been triggered by misaligned catchment IDs.

If you see a lot of auto ghost nodes after export (double or triple figures), then something is seriously wrong in your project and there is little point even trying to upload these shapefiles to GDMS.

The GDMS export routine creates two files for each catchment in the project and this is exactly the reason why it is essential to keep a solid grip on the job data within the WinCan project at all times during data processing and site analysis:

• A zipped folder containing all of the GDMS data files for upload including the inspection photos, shapefiles and observation .dbf file. You can try to upload this file to GDMS and if the data is good, then it will be approved.

• A .dxf file containing the shape geometry of the project including attribute data, but without any significant layer styling. This is not a GDMS deliverable which is why it is saved outside of the .zip file, but it creates a geometrically correct representation of the data that is being uploaded in a generic layered CAD file from where the user can style and design their own CAD files for delivery to the area client.

The upload process to GDMS is not for any real description here, because that is a GDMS process using their systems and website, but it is worth just noting a few minor points that are maybe a little unclear with their system:

• If you have ‘Drainage Checker’ rights on the system for the area that you are working on, then you will be permitted to carry out a ‘test upload'.

  • This means that you can upload the zip folder that is exported from WinCan VX into the GDMS system against the relevant scheme reference.

  • Usually, a few minutes after upload, you will be able to download a .csv file which gives a list of critical and non-critical errors about your data.

  • If there are no critical errors, then the upload can be considered successful, but the data may still have a great many non-critical errors.

  • The validation rules that are used in WinCan VX are about 95% aligned with the validation rules that are applied on the GDMS system, and Errors in VX align with ‘critical’ errors on GDMS, while Warnings in VX align with ‘non-critical' errors in GDMS.

  • You should bare in mind at all times, that when you upload a zipped set of data to GDMS for validation, it is the shapefiles that are being validated and not your WinCan VX database project, and during the export process from WinCan VX to shapefile, there are a number of automatic edits and data cleanup actions that happen in the background, so the data that is in the shapefile does not necessarily match exactly with the data that is in your VX project. However, if there are errors on upload, they can always be fixed in VX, re-exported and try again.

  • There is a 2nd validation tool available from WinCan VX v10.0 onwards which is contained with WinCan Map, and the rules in here are 100% aligned with the GDMS validation rules. The reason for this is that this tool is validating the shapefiles and not the VX database. More information on how to use this tool can be found in Consolidation ToolsPreview

• Access to ‘Drainage Checker’ status on the GDMS system can only be given by the areas, so you should discuss this with them directly if you need it. You can request such access by clicking here.

• If you have a successful upload with no critical errors, then there is a possibility that you may see some further warnings about ‘Possible Data Loss’. Do not be alarmed by this.

  • The fact is, the upload test has been successful, and that’s the bottom line really.

  • This ‘possible data loss’ warning is triggered when any asset that had condition data (i.e. observations) before has been removed from the scheme (i.e. it has been deleted from the VX project. Unfortunately, the warning does not inform you which asset is being referred to, so if your project has 10,000 assets in it and 1 is marked as a warning, then finding it can be looking for a needle in a haystack. But, it is nonetheless possible with some smart thinking with Excel spreadsheets directly from the original shapefiles that were imported and the Report Generator outputs from WinCan VX.

  • From Mott MacDonald - It’s just a final sense check for the person uploading to say, “Some data has been deleted, is that expected?”.  If Yes, the client ignores it and just import the data but if No, then they go to the contractor and say “Why has some data been deleted?”.

Back to the top.