...
| Table of Contents | ||||
|---|---|---|---|---|
|
...
Export Hierarchy
There are a number of things to consider that all affect the results that are output from WinCan VX during the export process, and they are almost all handled automatically in the background during the data exchange routine without any user options available to the user.
...
In the HADDMS data files, there is only ever one combined inspection and asset data record of each object, but of course, WinCan VX has the concept of ‘multiple inspections’ and ‘combined inspections’, so how do we decide which inspection to export when there are several to chose from? This is where the Export Hierarchy comes into play, and the selected export order of the one inspection record to be exported (applies to all object types) is like this:
Combined inspection - if a combined inspection record exists on an object then it is exported and no other inspection data is exported for the current asset.
A complete inspection from end to end - this might not be the most recent inspection because maybe there was am abandoned inspection carried out after the complete one, but this does not matter under HADDMS rules - a complete inspection trumps an abandoned inspection regardless of date and time.
An abandoned inspection - the most recent abandoned inspection will be exported. Of course if there was at least one abandoned inspection from each end then there should be a combined inspection, so we are back up this tree to option 1.
The original imported data, untouched, unvalidated, not re-scored and unmodified.
At the end of this export process, every asset has an inspection record exported into the data regardless of whether it was inspected by the contractor during this package of work or not.
...
Creating HADDMS Exports
The files that are produced by the HADDMS export routine are described in the next section, but are essentially only the files required by HADDMS for upload and a .dxf file. The important part about this is that any other deliverables that your client may like to receive have to be created separately from the export routine.
...
Before running the export routine, the user is strongly advised to revisit and evaluate the following checklist in the order given and ensure that all items are 100% satisfied, because failure to do this will almost certainly result in critical rejection errors when the shapefiles are attempted to be uploaded to HADDMS:
Run the ‘Duplicates Finder’ tool on Sections and Nodes from the Edit ribbon (Tools → Misc Tools list before VX v15.2) and resolve issues.
Execute the ‘Delete Empty Inspections’ tool from the Edit ribbon (Tools → Misc Tools list before VX v15.2) list.
Execute the ‘Regenerate Sort order’ tool from the Edit ribbon (Tools → Misc Tools list before VX v15.2) list.
Run the inspection merging tool using the default options as described here.
Run the ‘Calculate Invert Levels’ tool on the project.
Run the ‘Job Checker’ tool from the Edit ribbon (Tools → Misc Tools list before VX v15.2) list and resolve issues.
In WinCan VX v10.3 or lower only, regenerate the section and lateral SUPP_REFs using the tool from the project tab.
Validate the Sections
...
and clear all Errors to zero.
Validate the Laterals
...
and clear all Errors to zero.
Validate the Nodes
...
and clear all Errors to zero.
The first step in the export process is to click on the ‘Export’ button in the ‘Data Exchange’ ribbon:
...
When the process has finished, you maybe presented with some warnings, but provided the final message says that the ‘Process has been finished’, then you are good to go, and clicking on the folder icon to the right will take you straight to the export file folder ([Project Folder] → Misc → Exchange → [Project Name]_Date_Time and the folder containing the export files will be named [Project Name]_DDMS_CD535. Inside here, you will find the export files as described:
...
Export final stage.
...
Auto Nodes
A common question after export is ‘Why has my data now got Auto ghost nodes in it after export?’
There are 2 drivers of auto created nodes inside the data after the export routine has been executed. The first is clearly visible in the WinCan project after export, and the second is only visible in the shapefiles with no change in the project, although the trigger for the change will be clear.
Auto_GN_nnnn
After export of the data to the HADDMS shapefiles, you may see some additional nodes in your WinCan project named Auto_GN_nnnn where nnnn is a numeric counter.
These have been created automatically by the export routine to satisfy one of the ‘golden rules' of HADDMS data for a successful upload. This rule is that every continuous item must have a point item at each end that is in the same catchment scheme.
Imagine this scenario:
...
Points and continuous items in mixed up catchments.
It is not possible to enter data into HADDMS like this, because Pipe 2 has one node in catchment 1 and one node in catchment 2.
So, if you have a section that is in catchment 1 and it has a node at each end where one node is also in catchment 1, but the other node is in catchment 2, then a Auto_GN node will be created in catchment 1 and linked to the section to satisfy this golden rule. The new Auto node will be a Ghost Node at the same coordinates as the original node, so now you have 2 nodes on top of each other in the same place, like this:
...
How it looks after export.
The drawing has been exploded to demonstrate what has happened here, but you should read this so that that Manhole 2 and Auto Ghost Node 1 are at exactly the same position - they are on top of each other.
Now the data is good to go to HADDMS, because Pipe 1 has a node at each end that is in the same catchment 1 as the continuous item, and Pipe 2 also has a node at each end that is in the same catchment 2 as the continuous item.
This feature is somewhat complicated by the fact that only inspections are linked to catchments, not assets. So, on investigation, you might have:
A section with multiple inspections - you must check that all of the inspections of the section are in the correct catchment and make amendments if necessary.
Nodes connected to the section which might also have multiple inspections - you must check that all of the inspections of the nodes connected to the section are in the same catchment as the section.
There is a tool available in WinCan VX 10.0 and higher which helps to resolve these issues before export:
...
The Job Checker tool.
When launched:
The application checks that all section inspections are in the same catchment as the original imported catchment from the very beginning of the process.
If they are not, then it automatically assigns the original read-only catchment to all inspections of the section.
It then goes on to do the same thing for all the inspections of the 2 nodes that are attached to the section, so the catchment information is auto-fixed in this scenario.
If the section is a newly added section that did not exist in the original data, then it checks that all inspections of the section are in the same catchment.
If they are not, then the user is prompted to select a catchment from the available list for the section.
Then, the inspections of the nodes connected to the section are moved to this same catchment also.
A common user error when checking the cohesion of section catchment IDs against its node catchment IDs is to not consider multiple inspections - it is crucially important to align the catchment ID of all inspections of the section first with the original catchment ID of the read-only inspection (if it exists), or with a user selected catchment if it is new data, then align all of node inspections of both nodes to the same catchment ID as the section.
Auto_SC/DC/LC/MC_nnnn
During the export process, there can be occasions when the the exported shapefiles have unexpected point items and extra continuous items that are not visible inside the WinCan VX project.
The reason that this happens is because section 2.26 of the CD535 documentation states:
2.26 Continuous assets shall be split into two separately recorded assets where a significant change in the asset occurs, such as a watershed, significant change in dimensions, materials or lining, or where there is an intermediate flow control device.
NOTE Where there is no physical asset at the location of a significant change, a ghost node is recorded, and is the upstream or downstream asset as appropriate of the two continuous assets.
What this means is that wherever there is a change of significant attribute in the inspection data, we must split the continuous item and create a ghost node connected to both items.
Significant changes of attribute are:
Shape change - triggered by any SCx observation code (not SC code). At this point, an auto ghost node will be created and named Auto_SC_nnnn indicating that this auto node has been created at a Shape Change in the asset.
Size change - triggered by the SC observation code. At this point, an auto ghost node will be created and named Auto_DC_nnnn indicating that this auto node has been created at a Dimension Change in the asset, and this will only be triggered when the change in the primary dimension is greater or equal to 75mm (i.e. one nominal pipe size in most cases).
Material change - triggered by any MCxxx observation code. At this point, an auto ghost node will be created and named Auto_MC_nnnn indicating that this auto node has been created at a Material Change in the asset.
Lining material change - triggered by any LCxxx observation code. At this point, an auto ghost node will be created and named Auto_LC_nnnn indicating that this auto node has been created at a Lining Change in the asset.
At the point of creating a new ghost node with this naming convention, the inspection data and continuous attribute data will also be split accordingly, so now there will be 2 continuous items where before there was only 1.
Also, the assets that have been split will be re-scored. This is the underlying driver behind this feature, because for some observation codes (consider the deformed code), the scoring is different for flexible and rigid materials, so if the material changes from say concrete to plastic and there is deformation in the pipe, we must split the pipe into 2 to get the correct grades for each part of the pipe.
| Info |
|---|
Notes: When these ‘hidden’ auto nodes are created, they are only created in the shapefile exports and there is no change to the data in the WinCan VX project. The reason for this is that there was only ever one video inspection of this pipe, and the operator simply created an inspection with some codes, as is good an proper. We only create one report in the pdf file for this inspection, there is only one video and there is only one record of the inspection in the project database, and this has to stay this way for our sanity inside the database. Hence, these are the hidden auto nodes that are only in the exported shapefiles. But, if you need to know where they have been created, then use the filtering options to find observation codes SC% MC% and LC% in the inspections in the project. These are the places where the these auto nodes will be created. Furthermore, if you are struggling to diagnose a HADDMS upload error based on these point items and the continuous items that are created as a result of this process, then there is little point in searching through the WinCan VX project to find these nodes, because they simply will not be there. They are only in the shapefiles. You should open the shapefiles directly in WinCan Map without WinCan VX, and start your investigations from there, which then may lead you into the WinCan VX project to fix the problems. Remember - whenever you run the export routine for the final deliverables, a backup is made of the current database inside [Project Folder] → Misc → Backup. We do this because we are modifying the data during the export process and you have no control over this process, so if you are not happy with the results after export, you can roll back your project to the place where it was at when you started the export process if you wish, make some fixes and run the export again. |
...
Data Review After Export
After running a HADDMS export, there will usually be changes that have been made to your project during the export routine. This has been mentioned already here. There will also be a backup made of you’re database files immediately before the export was run, so…
If you are unhappy with the database after export, you must immediately roll back the database files, investigate the issues and try again.
The most common issue with bad data after export is Auto_GN_nnnn nodes as described above. These auto nodes are created with good intention to help you deliver HADDMS compatible data, but it should only happen very rarely with ‘good’ data.
So, after export, check out your node list and if there are a lot of Auto_GN_nnnn nodes, then do not go any further. Roll back the database, investigate and fix the issues. The issue will most likely be linked to the misalignment of Catchment IDs between the continuous item inspections and their two linked point item inspections.
How many is ‘a lot’ ? There is no fixed answer to this, but in a perfect world, there will be zero auto ghost nodes created during export. As an example, if you imported 2 HADDMS catchments into VX and then just exported them again, there will be no auto ghost nodes created, because there will have been no misalignment of the continuous-point relationship in the imported data.
But, when working on site, we modify and update the data, and we add new assets, so this is where genuine discrepancies may creep in to the project.
As a rule of thumb, the number of auto ghost nodes created should be in single figures, and very close to zero. Regardless of how many auto nodes are created, they should all be investigated closely to see if they really are genuine or if they have been triggered by misaligned catchment IDs.
If you see a lot of auto ghost nodes after export (double or triple figures), then something is seriously wrong in your project and there is little point even trying to upload these shapefiles to HADDMS.
...
Understanding the Outputs
The HADDMS export routine creates two files for each catchment in the project and this is exactly the reason why it is essential to keep a solid grip on the job data within the WinCan project at all times during data processing and site analysis:
A zipped folder containing all of the HADDMS data files for upload including the inspection photos, shapefiles and observation .dbf file. You can try to upload this file to HADDMS and if the data is good, then it will be approved.
A .dxf file containing the shape geometry of the project including attribute data, but without any significant layer styling. This is not a HADDMS deliverable which is why it is saved outside of the .zip file, but it creates a geometrically correct representation of the data that is being uploaded in a generic layered CAD file from where the user can style and design their own CAD files for delivery to the area client.
...
Uploading to HADDMS
The upload process to HADDMS is not for any real description here, because that is a HADDMS process using their systems and website, but it is worth just noting a few minor points that are maybe a little unclear with their system:
If you have ‘Drainage Checker’ rights on the system for the area that you are working on, then you will be permitted to carry out a ‘test upload'.
This means that you can upload the zip folder that is exported from WinCan VX into the HADDMS system against the relevant scheme reference.
Usually, a few minutes after upload, you will be able to download a .csv file which gives a list of critical and non-critical errors about your data.
If there are no critical errors, then the upload can be considered successful, but the data may still have a great many non-critical errors.
The validation rules that are used in WinCan VX are about 95% aligned with the validation rules that are applied on the HADDMS system, and Errors in VX align with ‘critical’ errors on HADDMS, while Warnings in VX align with ‘non-critical' errors in HADDMS.
You should bare in mind at all times, that when you upload a zipped set of data to HADDMS for validation, it is the shapefiles that are being validated and not your WinCan VX database project, and during the export process from WinCan VX to shapefile, there are a number of automatic edits and data cleanup actions that happen in the background, so the data that is in the shapefile does not necessarily match exactly with the data that is in your VX project. However, if there are errors on upload, they can always be fixed in VX, re-exported and try again.
There is a 2nd validation tool available from WinCan VX v10.0 onwards which is contained with WinCan Map, and the rules in here are 100% aligned with the HADDMS validation rules. The reason for this is that this tool is validating the shapefiles and not the VX database. More information on how to use this tool can be found in Consolidation Tools
Access to ‘Drainage Checker’ status on the HADDMS system can only be given by the areas, so you should discuss this with them directly if you need it. You can request such access by clicking here.
If you have a successful upload with no critical errors, then there is a possibility that you may see some further warnings about ‘Possible Data Loss’. Do not be alarmed by this.
The fact is, the upload test has been successful, and that’s the bottom line really.
This ‘possible data loss’ warning is triggered when any asset that had condition data (i.e. observations) before has been removed from the scheme (i.e. it has been deleted from the VX project. Unfortunately, the warning does not inform you which asset is being referred to, so if your project has 10,000 assets in it and 1 is marked as a warning, then finding it can be looking for a needle in a haystack. But, it is nonetheless possible with some smart thinking with Excel spreadsheets directly from the original shapefiles that were imported and the Report Generator outputs from WinCan VX.
From Mott MacDonald - It’s just a final sense check for the person uploading to say, “Some data has been deleted, is that expected?”. If Yes, the client ignores it and just import the data but if No, then they go to the contractor and say “Why has some data been deleted?”.
Example of how this might look on your HADDMS screen after upload:
...
Example of a ‘Possible Data Loss’ warning.
...