...
| Table of Contents | ||||
|---|---|---|---|---|
|
...
Export Hierarchy
There are a number of things to consider that all affect the results that are output from WinCan VX during the export process, and they are almost all handled automatically in the background during the data exchange routine without any user options available to the user.
...
In the HADDMS data files, there is only ever one combined inspection and asset data record of each object, but of course, WinCan VX has the concept of ‘multiple inspections’ and ‘combined inspections’, so how do we decide which inspection to export when there are several to chose from? This is where the Export Hierarchy comes into play, and the selected export order of the one inspection record to be exported (applies to all object types) is like this:
Combined inspection - if a combined inspection record exists on an object then it is exported and no other inspection data is exported for the current asset.
A complete inspection from end to end - this might not be the most recent inspection because maybe there was am abandoned inspection carried out after the complete one, but this does not matter under HADDMS rules - a complete inspection trumps an abandoned inspection regardless of date and time.
An abandoned inspection - the most recent abandoned inspection will be exported. Of course if there was at least one abandoned inspection from each end then there should be a combined inspection, so we are back up this tree to option 1.
The original imported data, untouched, unvalidated, not re-scored and unmodified.
At the end of this export process, every asset has an inspection record exported into the data regardless of whether it was inspected by the contractor during this package of work or not.
...
Creating HADDMS Exports
The files that are produced by the HADDMS export routine are described in the next section, but are essentially only the files required by HADDMS for upload and a .dxf file. The important part about this is that any other deliverables that your client may like to receive have to be created separately from the export routine.
...
Before running the export routine, the user is strongly advised to revisit and evaluate the following checklist in the order given and ensure that all items are 100% satisfied, because failure to do this will almost certainly result in critical rejection errors when the shapefiles are attempted to be uploaded to HADDMS:
Run the ‘Duplicates Finder’ tool on Sections and Nodes from the Edit ribbon (Tools → Misc Tools list before VX v15.2) and resolve issues.
Execute the ‘Delete Empty Inspections’ tool from the Edit ribbon (Tools → Misc Tools list before VX v15.2) list.
Execute the ‘Regenerate Sort order’ tool from the Edit ribbon (Tools → Misc Tools list before VX v15.2) list.
Run the inspection merging tool using the default options as described here.
Run the ‘Calculate Invert Levels’ tool on the project.
Run the ‘Job Checker’ tool from the Edit ribbon (Tools → Misc Tools list before VX v15.2) list and resolve issues.
In WinCan VX v10.3 or lower only, regenerate the section and lateral SUPP_REFs using the tool from the project tab.
Validate the Sections
...
and clear all Errors to zero.
Validate the Laterals
...
and clear all Errors to zero.
Validate the Nodes
...
and clear all Errors to zero.
The first step in the export process is to click on the ‘Export’ button in the ‘Data Exchange’ ribbon:
...
When the process has finished, you maybe presented with some warnings, but provided the final message says that the ‘Process has been finished’, then you are good to go, and clicking on the folder icon to the right will take you straight to the export file folder ([Project Folder] → Misc → Exchange → [Project Name]_Date_Time and the folder containing the export files will be named [Project Name]_DDMS_CD535. Inside here, you will find the export files as described:
...
Export final stage.
...
Auto Nodes
A common question after export is ‘Why has my data now got Auto ghost nodes in it after export?’
...
These have been created automatically by the export routine to satisfy one of the ‘golden rules' of HADDMS data for a successful upload. This rule is that every continuous item must have a point item at each end that is in the same catchment scheme.So, if you have a section that is in catchment
Imagine this scenario:
...
Points and continuous items in mixed up catchments.
It is not possible to enter data into HADDMS like this, because Pipe 2 has one node in catchment 1 and one node in catchment 2.
So, if you have a section that is in catchment 1 and it has a node at each end where one node is also in catchment 1, but the other node is in catchment 2, then a Auto_GN node will be created in catchment 1 and linked to the section to satisfy this golden rule. The new Auto node will be a Ghost Node at the same coordinates as the original node, so now you have 2 nodes on top of each other in the same place, but only one of them is valid.
This feature is somewhat complicated by the fact that only inspections are linked to catchments, not assets. So, on investigation, you might have:
A section with multiple inspections - you must check that all of the inspections of the section are in the correct catchment and make amendments if necessary.
Nodes connected to the section which might also have multiple inspections - you must check that all of the inspections of the nodes connected to the section are in the same catchment as the section.
There is a tool available in WinCan VX 10.0 and higher which helps to resolve these issues before export:
...
The Job Checker tool.
When launched:
The application checks that all section inspections are in the same catchment as the original imported catchment from the very beginning of the process.
If they are not, then it automatically assigns the original read-only catchment to all inspections of the section.
It then goes on to do the same thing for all the inspections of the 2 nodes that are attached to the section, so the catchment information is auto-fixed in this scenario.
If the section is a newly added section that did not exist in the original data, then it checks that all inspections of the section are in the same catchment.
If they are not, then the user is prompted to select a catchment from the available list for the section.
Then, the inspections of the nodes connected to the section are moved to this same catchment also.
...
Warning - at the time of writing this (Prod v10.1), there is a bug in the pop up window that appears after launching the tool.
The list is showing the nodes that are in the wrong catchments, and not the sections. However, it does indicate where corrections need to be made in the data, but as a rule-of-thumb, the user must align all the node catchments with the section catchments.
...
like this:
...
How it looks after export.
The drawing has been exploded to demonstrate what has happened here, but you should read this so that that Manhole 2 and Auto Ghost Node 1 are at exactly the same position - they are on top of each other.
Now the data is good to go to HADDMS, because Pipe 1 has a node at each end that is in the same catchment 1 as the continuous item, and Pipe 2 also has a node at each end that is in the same catchment 2 as the continuous item.
This feature is somewhat complicated by the fact that only inspections are linked to catchments, not assets. So, on investigation, you might have:
A section with multiple inspections - you must check that all of the inspections of the section are in the correct catchment and make amendments if necessary.
Nodes connected to the section which might also have multiple inspections - you must check that all of the inspections of the nodes connected to the section are in the same catchment as the section.
There is a tool available in WinCan VX 10.0 and higher which helps to resolve these issues before export:
...
The Job Checker tool.
When launched:
The application checks that all section inspections are in the same catchment as the original imported catchment from the very beginning of the process.
If they are not, then it automatically assigns the original read-only catchment to all inspections of the section.
It then goes on to do the same thing for all the inspections of the 2 nodes that are attached to the section, so the catchment information is auto-fixed in this scenario.
If the section is a newly added section that did not exist in the original data, then it checks that all inspections of the section are in the same catchment.
If they are not, then the user is prompted to select a catchment from the available list for the section.
Then, the inspections of the nodes connected to the section are moved to this same catchment also.
A common user error when checking the cohesion of section catchment IDs against its node catchment IDs is to not consider multiple inspections - it is crucially important to align the catchment ID of all inspections of the section first with the original catchment ID of the read-only inspection (if it exists), or with a user selected catchment if it is new data, then align all of node inspections of both nodes to the same catchment ID as the section.
...
| Info |
|---|
Notes: When these ‘hidden’ auto nodes are created, they are only created in the shapefile exports and there is no change to the data in the WinCan VX project. The reason for this is that there was only ever one video inspection of this pipe, and the operator simply created an inspection with some codes, as is good an proper. We only create one report in the pdf file for this inspection, there is only one video and there is only one record of the inspection in the project database, and this has to stay this way for our sanity inside the database. Hence, these are the hidden auto nodes that are only in the exported shapefiles. But, if you need to know where they have been created, then use the filtering options to find observation codes SC% MC% and LC% in the inspections in the project. These are the places where the these auto nodes will be created. Furthermore, if you are struggling to diagnose a HADDMS upload error based on these point items and the continuous items that are created as a result of this process, then there is little point in searching through the WinCan VX project to find these nodes, because they simply will not be there. They are only in the shapefiles. You should open the shapefiles directly in WinCan Map without WinCan VX, and start your investigations from there, which then may lead you into the WinCan VX project to fix the problems. Remember - whenever you run the export routine for the final deliverables, a backup is made of the current database inside [Project Folder] → Misc → Backup. We do this because we are modifying the data during the export process and you have no control over this process, so if you are not happy with the results after export, you can roll back your project to the place where it was at when you started the export process if you wish, make some fixes and run the export again. |
...
Data Review After Export
After running a HADDMS export, there will usually be changes that have been made to your project during the export routine. This has been mentioned already here. There will also be a backup made of you’re database files immediately before the export was run, so…
...
If you see a lot of auto ghost nodes after export (double or triple figures), then something is seriously wrong in your project and there is little point even trying to upload these shapefiles to HADDMS.
...
Understanding the Outputs
The HADDMS export routine creates two files for each catchment in the project and this is exactly the reason why it is essential to keep a solid grip on the job data within the WinCan project at all times during data processing and site analysis:
A zipped folder containing all of the HADDMS data files for upload including the inspection photos, shapefiles and observation .dbf file. You can try to upload this file to HADDMS and if the data is good, then it will be approved.
A .dxf file containing the shape geometry of the project including attribute data, but without any significant layer styling. This is not a HADDMS deliverable which is why it is saved outside of the .zip file, but it creates a geometrically correct representation of the data that is being uploaded in a generic layered CAD file from where the user can style and design their own CAD files for delivery to the area client.
...
Uploading to HADDMS
The upload process to HADDMS is not for any real description here, because that is a HADDMS process using their systems and website, but it is worth just noting a few minor points that are maybe a little unclear with their system:
If you have ‘Drainage Checker’ rights on the system for the area that you are working on, then you will be permitted to carry out a ‘test upload'.
This means that you can upload the zip folder that is exported from WinCan VX into the HADDMS system against the relevant scheme reference.
Usually, a few minutes after upload, you will be able to download a .csv file which gives a list of critical and non-critical errors about your data.
If there are no critical errors, then the upload can be considered successful, but the data may still have a great many non-critical errors.
The validation rules that are used in WinCan VX are about 95% aligned with the validation rules that are applied on the HADDMS system, and Errors in VX align with ‘critical’ errors on HADDMS, while Warnings in VX align with ‘non-critical' errors in HADDMS.
You should bare in mind at all times, that when you upload a zipped set of data to HADDMS for validation, it is the shapefiles that are being validated and not your WinCan VX database project, and during the export process from WinCan VX to shapefile, there are a number of automatic edits and data cleanup actions that happen in the background, so the data that is in the shapefile does not necessarily match exactly with the data that is in your VX project. However, if there are errors on upload, they can always be fixed in VX, re-exported and try again.
There is a 2nd validation tool available from WinCan VX v10.0 onwards which is contained with WinCan Map, and the rules in here are 100% aligned with the HADDMS validation rules. The reason for this is that this tool is validating the shapefiles and not the VX database. More information on how to use this tool can be found in Consolidation Tools
Access to ‘Drainage Checker’ status on the HADDMS system can only be given by the areas, so you should discuss this with them directly if you need it. You can request such access by clicking here.
If you have a successful upload with no critical errors, then there is a possibility that you may see some further warnings about ‘Possible Data Loss’. Do not be alarmed by this.
The fact is, the upload test has been successful, and that’s the bottom line really.
This ‘possible data loss’ warning is triggered when any asset that had condition data (i.e. observations) before has been removed from the scheme (i.e. it has been deleted from the VX project. Unfortunately, the warning does not inform you which asset is being referred to, so if your project has 10,000 assets in it and 1 is marked as a warning, then finding it can be looking for a needle in a haystack. But, it is nonetheless possible with some smart thinking with Excel spreadsheets directly from the original shapefiles that were imported and the Report Generator outputs from WinCan VX.
From Mott MacDonald - It’s just a final sense check for the person uploading to say, “Some data has been deleted, is that expected?”. If Yes, the client ignores it and just import the data but if No, then they go to the contractor and say “Why has some data been deleted?”.
Example of how this might look on your HADDMS screen after upload:
...