Skip to content

S1201 schema fix and data refresh #2110

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
balit-raibot merged 24 commits into from
Aug 5, 2026
Merged

S1201 schema fix and data refresh #2110

Show file tree
Hide file tree
Changes from 10 commits
Commits
Show all changes
24 commits
Select commit Hold shift + click to select a range
63117d1
fixing download script
balit-raibot Jul 15, 2026
9908497
fixed download script
balit-raibot Jul 22, 2026
673a5ed
corrected schema
balit-raibot Jul 22, 2026
a41fe0c
Merge branch 'master' into s1201_schema_fix_and_data_refresh
balit-raibot Jul 22, 2026
f796294
fixed SVs
balit-raibot Jul 22, 2026
26075e8
Merge branch 's1201_schema_fix_and_data_refresh' of https://github.co…
balit-raibot Jul 22, 2026
470c4ee
fixed linting errors
balit-raibot Jul 22, 2026
b9d0ba0
fixed gemini comments
balit-raibot Jul 22, 2026
d31aa25
fixing linting errors
balit-raibot Jul 22, 2026
8ffdabd
removing log file
balit-raibot Jul 22, 2026
eb8e57f
fixed gemini comments
balit-raibot Jul 22, 2026
4c24274
fixing gemini comments
balit-raibot Jul 23, 2026
f9e149c
fixed linting errors
balit-raibot Jul 23, 2026
f2a9f6a
fixed linting errors
balit-raibot Jul 23, 2026
e8c4678
Merge branch 'master' into s1201_schema_fix_and_data_refresh
balit-raibot Jul 24, 2026
ce723fe
Merge branch 'master' into s1201_schema_fix_and_data_refresh
balit-raibot Jul 28, 2026
03fa691
reverting override removal
balit-raibot Jul 31, 2026
16ed0c5
Merge branch 's1201_schema_fix_and_data_refresh' of https://github.co…
balit-raibot Jul 31, 2026
7d45f9f
Merge branch 'master' into s1201_schema_fix_and_data_refresh
balit-raibot Jul 31, 2026
51a33b0
Merge branch 'master' into s1201_schema_fix_and_data_refresh
balit-raibot Aug 4, 2026
476696d
fixing cloudbuild failures
balit-raibot Aug 4, 2026
40a60a8
Merge branch 's1201_schema_fix_and_data_refresh' of https://github.co…
balit-raibot Aug 4, 2026
1f3b014
fixing cloudbuild failures
balit-raibot Aug 4, 2026
80b31c8
fixing cloudbuild failures
balit-raibot Aug 5, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions import-automation/executor/requirements.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
# Requirements for Python scripts in this repo that have automation enabled!

absl-py
aiolimiter
arcgis2geojson
beautifulsoup4
chardet
Expand Down
Empty file removed scripts/us_census/acs5yr/subject_tables/common/make_column_map.log
View file
Open in desktop
Empty file.
177 changes: 37 additions & 140 deletions scripts/us_census/acs5yr/subject_tables/s1201/S1201_spec.json
View file
Open in desktop

Large diffs are not rendered by default.

8 changes: 4 additions & 4 deletions scripts/us_census/acs5yr/subject_tables/s1201/generate_col_map.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,11 +25,11 @@
from collections import OrderedDict
import pandas as pd

# Allows the following module imports to work when running as a script
# Allows 'scripts' package to be imported when run as a script or module directly
_SCRIPT_PATH = os.path.dirname(os.path.abspath(__file__))
sys.path.append(os.path.join(_SCRIPT_PATH,
'../common')) # for statvar_dcid_generator
from generate_col_map import generate_stat_var_map
sys.path.append(os.path.abspath(os.path.join(_SCRIPT_PATH, '../../../../..')))
Comment thread
balit-raibot marked this conversation as resolved.
Outdated

from scripts.us_census.acs5yr.subject_tables.common.generate_col_map import generate_stat_var_map


def process_zip_file(zip_file_path,
Expand Down
28 changes: 17 additions & 11 deletions scripts/us_census/acs5yr/subject_tables/s1201/process.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,23 +12,27 @@
# See the License for the specific language governing permissions and
# limitations under the License.
"""Generic proces module to generate the csv/tmcf and csv"""
# TODO: Add unit tests
# Allows the sibling/parent module imports to work when running as a script or module
import os
import sys
_SCRIPT_PATH = os.path.dirname(os.path.abspath(__file__))
_curr = _SCRIPT_PATH
while _curr and _curr != os.path.dirname(_curr):
if os.path.exists(os.path.join(_curr, '.git')) or os.path.exists(os.path.join(_curr, 'WORKSPACE')):
sys.path.append(_curr)
break
_curr = os.path.dirname(_curr)

# TODO: Add unit tests
import json
from zipfile import ZipFile
import pandas as pd

from absl import app, flags
# TODO: logs from the column map step is empty when invoked from here, needs to be checked

from .generate_col_map import generate_stat_var_map, process_zip_file

# Allows the following module imports to work when running as a script
_SCRIPT_PATH = os.path.dirname(os.path.abspath(__file__))
sys.path.append(os.path.join(_SCRIPT_PATH,
'../common')) # for col_map_generator, data_loader
from data_loader import process_subject_tables
from scripts.us_census.acs5yr.subject_tables.s1201.generate_col_map import process_zip_file
from scripts.us_census.acs5yr.subject_tables.common.data_loader import process_subject_tables

FLAGS = flags.FLAGS
flags.DEFINE_string(
Expand Down Expand Up @@ -56,9 +60,9 @@ def set_column_map(input_path, spec_path, output_dir):
generated_col_map = process_zip_file(input_path,
spec_path,
write_output=False)
f = open(os.path.join(output_dir, 'column_map.json'), 'w')
json.dump(generated_col_map, f, indent=4)
f.close()
os.makedirs(output_dir, exist_ok=True)
Comment thread
balit-raibot marked this conversation as resolved.
Outdated
with open(os.path.join(output_dir, 'column_map.json'), 'w') as f:
json.dump(generated_col_map, f, indent=4)
Comment thread
balit-raibot marked this conversation as resolved.


def main(argv):
Expand All @@ -71,6 +75,8 @@ def main(argv):
has_percent = FLAGS.has_percent
debug = FLAGS.debug

os.makedirs(output_dir, exist_ok=True)

# TODO: remove the constraint of inputs being only zip file
# context: the current implementation of the column map generator accepts
# only zip files as input and we will need to add new methods to handle inputs
Expand Down
600 changes: 300 additions & 300 deletions scripts/us_census/acs5yr/subject_tables/s1201/testdata/S1201_cleaned.csv
View file
Open in desktop

Large diffs are not rendered by default.

60 changes: 30 additions & 30 deletions scripts/us_census/acs5yr/subject_tables/s1201/testdata/S1201_output.mcf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -464,23 +464,23 @@ gender: dcs:Female
employmentStatus: dcs:BLS_InLaborForce
typeOf: dcs:StatisticalVariable

Node: dcid:Count_Person_MarriedAndNotSeparated
Node: dcid:Count_Person_15OrMoreYears_MarriedAndNotSeparated
statType: dcs:measuredValue
measuredProperty: dcs:count
age: [15 - Years]
populationType: dcs:Person
maritalStatus: dcs:MarriedAndNotSeparated
typeOf: dcs:StatisticalVariable

Node: dcid:MarginOfError_Count_Person_MarriedAndNotSeparated
Node: dcid:MarginOfError_Count_Person_15OrMoreYears_MarriedAndNotSeparated
statType: dcs:marginOfError
measuredProperty: dcs:count
age: [15 - Years]
populationType: dcs:Person
maritalStatus: dcs:MarriedAndNotSeparated
typeOf: dcs:StatisticalVariable

Node: dcid:Count_Person_Male_MarriedAndNotSeparated
Node: dcid:Count_Person_15OrMoreYears_Male_MarriedAndNotSeparated
statType: dcs:measuredValue
measuredProperty: dcs:count
age: [15 - Years]
Expand All @@ -489,7 +489,7 @@ maritalStatus: dcs:MarriedAndNotSeparated
gender: dcs:Male
typeOf: dcs:StatisticalVariable

Node: dcid:MarginOfError_Count_Person_Male_MarriedAndNotSeparated
Node: dcid:MarginOfError_Count_Person_15OrMoreYears_Male_MarriedAndNotSeparated
statType: dcs:marginOfError
measuredProperty: dcs:count
age: [15 - Years]
Expand Down Expand Up @@ -606,7 +606,7 @@ maritalStatus: dcs:MarriedAndNotSeparated
gender: dcs:Male
typeOf: dcs:StatisticalVariable

Node: dcid:Count_Person_Female_MarriedAndNotSeparated
Node: dcid:Count_Person_15OrMoreYears_Female_MarriedAndNotSeparated
statType: dcs:measuredValue
measuredProperty: dcs:count
age: [15 - Years]
Expand All @@ -615,7 +615,7 @@ maritalStatus: dcs:MarriedAndNotSeparated
gender: dcs:Female
typeOf: dcs:StatisticalVariable

Node: dcid:MarginOfError_Count_Person_Female_MarriedAndNotSeparated
Node: dcid:MarginOfError_Count_Person_15OrMoreYears_Female_MarriedAndNotSeparated
statType: dcs:marginOfError
measuredProperty: dcs:count
age: [15 - Years]
Expand Down Expand Up @@ -988,23 +988,23 @@ gender: dcs:Female
employmentStatus: dcs:BLS_InLaborForce
typeOf: dcs:StatisticalVariable

Node: dcid:Count_Person_Widowed
Node: dcid:Count_Person_15OrMoreYears_Widowed
statType: dcs:measuredValue
measuredProperty: dcs:count
age: [15 - Years]
populationType: dcs:Person
maritalStatus: dcs:Widowed
typeOf: dcs:StatisticalVariable

Node: dcid:MarginOfError_Count_Person_Widowed
Node: dcid:MarginOfError_Count_Person_15OrMoreYears_Widowed
statType: dcs:marginOfError
measuredProperty: dcs:count
age: [15 - Years]
populationType: dcs:Person
maritalStatus: dcs:Widowed
typeOf: dcs:StatisticalVariable

Node: dcid:Count_Person_Male_Widowed
Node: dcid:Count_Person_15OrMoreYears_Male_Widowed
statType: dcs:measuredValue
measuredProperty: dcs:count
age: [15 - Years]
Expand All @@ -1013,7 +1013,7 @@ maritalStatus: dcs:Widowed
gender: dcs:Male
typeOf: dcs:StatisticalVariable

Node: dcid:MarginOfError_Count_Person_Male_Widowed
Node: dcid:MarginOfError_Count_Person_15OrMoreYears_Male_Widowed
statType: dcs:marginOfError
measuredProperty: dcs:count
age: [15 - Years]
Expand Down Expand Up @@ -1130,7 +1130,7 @@ maritalStatus: dcs:Widowed
gender: dcs:Male
typeOf: dcs:StatisticalVariable

Node: dcid:Count_Person_Female_Widowed
Node: dcid:Count_Person_15OrMoreYears_Female_Widowed
statType: dcs:measuredValue
measuredProperty: dcs:count
age: [15 - Years]
Expand All @@ -1139,7 +1139,7 @@ maritalStatus: dcs:Widowed
gender: dcs:Female
typeOf: dcs:StatisticalVariable

Node: dcid:MarginOfError_Count_Person_Female_Widowed
Node: dcid:MarginOfError_Count_Person_15OrMoreYears_Female_Widowed
statType: dcs:marginOfError
measuredProperty: dcs:count
age: [15 - Years]
Expand Down Expand Up @@ -1512,23 +1512,23 @@ gender: dcs:Female
employmentStatus: dcs:BLS_InLaborForce
typeOf: dcs:StatisticalVariable

Node: dcid:Count_Person_Divorced
Node: dcid:Count_Person_15OrMoreYears_Divorced
statType: dcs:measuredValue
measuredProperty: dcs:count
age: [15 - Years]
populationType: dcs:Person
maritalStatus: dcs:Divorced
typeOf: dcs:StatisticalVariable

Node: dcid:MarginOfError_Count_Person_Divorced
Node: dcid:MarginOfError_Count_Person_15OrMoreYears_Divorced
statType: dcs:marginOfError
measuredProperty: dcs:count
age: [15 - Years]
populationType: dcs:Person
maritalStatus: dcs:Divorced
typeOf: dcs:StatisticalVariable

Node: dcid:Count_Person_Male_Divorced
Node: dcid:Count_Person_15OrMoreYears_Male_Divorced
statType: dcs:measuredValue
measuredProperty: dcs:count
age: [15 - Years]
Expand All @@ -1537,7 +1537,7 @@ maritalStatus: dcs:Divorced
gender: dcs:Male
typeOf: dcs:StatisticalVariable

Node: dcid:MarginOfError_Count_Person_Male_Divorced
Node: dcid:MarginOfError_Count_Person_15OrMoreYears_Male_Divorced
statType: dcs:marginOfError
measuredProperty: dcs:count
age: [15 - Years]
Expand Down Expand Up @@ -1654,7 +1654,7 @@ maritalStatus: dcs:Divorced
gender: dcs:Male
typeOf: dcs:StatisticalVariable

Node: dcid:Count_Person_Female_Divorced
Node: dcid:Count_Person_15OrMoreYears_Female_Divorced
statType: dcs:measuredValue
measuredProperty: dcs:count
age: [15 - Years]
Expand All @@ -1663,7 +1663,7 @@ maritalStatus: dcs:Divorced
gender: dcs:Female
typeOf: dcs:StatisticalVariable

Node: dcid:MarginOfError_Count_Person_Female_Divorced
Node: dcid:MarginOfError_Count_Person_15OrMoreYears_Female_Divorced
statType: dcs:marginOfError
measuredProperty: dcs:count
age: [15 - Years]
Expand Down Expand Up @@ -2036,23 +2036,23 @@ gender: dcs:Female
employmentStatus: dcs:BLS_InLaborForce
typeOf: dcs:StatisticalVariable

Node: dcid:Count_Person_Separated
Node: dcid:Count_Person_15OrMoreYears_Separated
statType: dcs:measuredValue
measuredProperty: dcs:count
age: [15 - Years]
populationType: dcs:Person
maritalStatus: dcs:Separated
typeOf: dcs:StatisticalVariable

Node: dcid:MarginOfError_Count_Person_Separated
Node: dcid:MarginOfError_Count_Person_15OrMoreYears_Separated
statType: dcs:marginOfError
measuredProperty: dcs:count
age: [15 - Years]
populationType: dcs:Person
maritalStatus: dcs:Separated
typeOf: dcs:StatisticalVariable

Node: dcid:Count_Person_Male_Separated
Node: dcid:Count_Person_15OrMoreYears_Male_Separated
statType: dcs:measuredValue
measuredProperty: dcs:count
age: [15 - Years]
Expand All @@ -2061,7 +2061,7 @@ maritalStatus: dcs:Separated
gender: dcs:Male
typeOf: dcs:StatisticalVariable

Node: dcid:MarginOfError_Count_Person_Male_Separated
Node: dcid:MarginOfError_Count_Person_15OrMoreYears_Male_Separated
statType: dcs:marginOfError
measuredProperty: dcs:count
age: [15 - Years]
Expand Down Expand Up @@ -2178,7 +2178,7 @@ maritalStatus: dcs:Separated
gender: dcs:Male
typeOf: dcs:StatisticalVariable

Node: dcid:Count_Person_Female_Separated
Node: dcid:Count_Person_15OrMoreYears_Female_Separated
statType: dcs:measuredValue
measuredProperty: dcs:count
age: [15 - Years]
Expand All @@ -2187,7 +2187,7 @@ maritalStatus: dcs:Separated
gender: dcs:Female
typeOf: dcs:StatisticalVariable

Node: dcid:MarginOfError_Count_Person_Female_Separated
Node: dcid:MarginOfError_Count_Person_15OrMoreYears_Female_Separated
statType: dcs:marginOfError
measuredProperty: dcs:count
age: [15 - Years]
Expand Down Expand Up @@ -2560,23 +2560,23 @@ gender: dcs:Female
employmentStatus: dcs:BLS_InLaborForce
typeOf: dcs:StatisticalVariable

Node: dcid:Count_Person_NeverMarried
Node: dcid:Count_Person_15OrMoreYears_NeverMarried
statType: dcs:measuredValue
measuredProperty: dcs:count
age: [15 - Years]
populationType: dcs:Person
maritalStatus: dcs:NeverMarried
typeOf: dcs:StatisticalVariable

Node: dcid:MarginOfError_Count_Person_NeverMarried
Node: dcid:MarginOfError_Count_Person_15OrMoreYears_NeverMarried
statType: dcs:marginOfError
measuredProperty: dcs:count
age: [15 - Years]
populationType: dcs:Person
maritalStatus: dcs:NeverMarried
typeOf: dcs:StatisticalVariable

Node: dcid:Count_Person_Male_NeverMarried
Node: dcid:Count_Person_15OrMoreYears_Male_NeverMarried
statType: dcs:measuredValue
measuredProperty: dcs:count
age: [15 - Years]
Expand All @@ -2585,7 +2585,7 @@ maritalStatus: dcs:NeverMarried
gender: dcs:Male
typeOf: dcs:StatisticalVariable

Node: dcid:MarginOfError_Count_Person_Male_NeverMarried
Node: dcid:MarginOfError_Count_Person_15OrMoreYears_Male_NeverMarried
statType: dcs:marginOfError
measuredProperty: dcs:count
age: [15 - Years]
Expand Down Expand Up @@ -2702,7 +2702,7 @@ maritalStatus: dcs:NeverMarried
gender: dcs:Male
typeOf: dcs:StatisticalVariable

Node: dcid:Count_Person_Female_NeverMarried
Node: dcid:Count_Person_15OrMoreYears_Female_NeverMarried
statType: dcs:measuredValue
measuredProperty: dcs:count
age: [15 - Years]
Expand All @@ -2711,7 +2711,7 @@ maritalStatus: dcs:NeverMarried
gender: dcs:Female
typeOf: dcs:StatisticalVariable

Node: dcid:MarginOfError_Count_Person_Female_NeverMarried
Node: dcid:MarginOfError_Count_Person_15OrMoreYears_Female_NeverMarried
statType: dcs:marginOfError
measuredProperty: dcs:count
age: [15 - Years]
Expand Down
Loading
Loading