[S] split: too many files
Closed, ResolvedPublicBUG REPORT
Actions

Assigned To

Authored By

	kostajh
	Oct 17 2023, 6:27 AM

Description

Steps to reproduce:

do a full import
the next day, run ./main.sh --updatedb true

➜ ./main.sh --updatedb true
Feed already exists. Aborting.
Feed already exists. Aborting.
Unzipping yesterday's file...
Sorting yesterday's file...
Unzipping today's file...
Aggregating today's properties...
Sorting today's file...
Finding lines unique to old file...
Finding lines unique to new file...
Processing unique files...
Joining and sorting unique files...
Outputting SQL statements...
{ changed: 7586731, removed: 2793173, inserted: 3126839 }
Concatenating all sql files...
Recording job details...
Importing properties generated by diff...
Resetting and generating substatement files...
split: too many files

Contents of $TMPDIR/sub:

query_split_aa query_split_dt query_split_hm query_split_lf query_split_oy query_split_sr query_split_wk
query_split_ab query_split_du query_split_hn query_split_lg query_split_oz query_split_ss query_split_wl
query_split_ac query_split_dv query_split_ho query_split_lh query_split_pa query_split_st query_split_wm
query_split_ad query_split_dw query_split_hp query_split_li query_split_pb query_split_su query_split_wn
query_split_ae query_split_dx query_split_hq query_split_lj query_split_pc query_split_sv query_split_wo
query_split_af query_split_dy query_split_hr query_split_lk query_split_pd query_split_sw query_split_wp
query_split_ag query_split_dz query_split_hs query_split_ll query_split_pe query_split_sx query_split_wq
query_split_ah query_split_ea query_split_ht query_split_lm query_split_pf query_split_sy query_split_wr
query_split_ai query_split_eb query_split_hu query_split_ln query_split_pg query_split_sz query_split_ws
query_split_aj query_split_ec query_split_hv query_split_lo query_split_ph query_split_ta query_split_wt
query_split_ak query_split_ed query_split_hw query_split_lp query_split_pi query_split_tb query_split_wu
query_split_al query_split_ee query_split_hx query_split_lq query_split_pj query_split_tc query_split_wv
query_split_am query_split_ef query_split_hy query_split_lr query_split_pk query_split_td query_split_ww
query_split_an query_split_eg query_split_hz query_split_ls query_split_pl query_split_te query_split_wx
query_split_ao query_split_eh query_split_ia query_split_lt query_split_pm query_split_tf query_split_wy
query_split_ap query_split_ei query_split_ib query_split_lu query_split_pn query_split_tg query_split_wz
query_split_aq query_split_ej query_split_ic query_split_lv query_split_po query_split_th query_split_xa
query_split_ar query_split_ek query_split_id query_split_lw query_split_pp query_split_ti query_split_xb
query_split_as query_split_el query_split_ie query_split_lx query_split_pq query_split_tj query_split_xc
query_split_at query_split_em query_split_if query_split_ly query_split_pr query_split_tk query_split_xd
query_split_au query_split_en query_split_ig query_split_lz query_split_ps query_split_tl query_split_xe
query_split_av query_split_eo query_split_ih query_split_ma query_split_pt query_split_tm query_split_xf
query_split_aw query_split_ep query_split_ii query_split_mb query_split_pu query_split_tn query_split_xg
query_split_ax query_split_eq query_split_ij query_split_mc query_split_pv query_split_to query_split_xh
query_split_ay query_split_er query_split_ik query_split_md query_split_pw query_split_tp query_split_xi
query_split_az query_split_es query_split_il query_split_me query_split_px query_split_tq query_split_xj
query_split_ba query_split_et query_split_im query_split_mf query_split_py query_split_tr query_split_xk
query_split_bb query_split_eu query_split_in query_split_mg query_split_pz query_split_ts query_split_xl
query_split_bc query_split_ev query_split_io query_split_mh query_split_qa query_split_tt query_split_xm
query_split_bd query_split_ew query_split_ip query_split_mi query_split_qb query_split_tu query_split_xn
query_split_be query_split_ex query_split_iq query_split_mj query_split_qc query_split_tv query_split_xo
query_split_bf query_split_ey query_split_ir query_split_mk query_split_qd query_split_tw query_split_xp
query_split_bg query_split_ez query_split_is query_split_ml query_split_qe query_split_tx query_split_xq
query_split_bh query_split_fa query_split_it query_split_mm query_split_qf query_split_ty query_split_xr
query_split_bi query_split_fb query_split_iu query_split_mn query_split_qg query_split_tz query_split_xs
query_split_bj query_split_fc query_split_iv query_split_mo query_split_qh query_split_ua query_split_xt
query_split_bk query_split_fd query_split_iw query_split_mp query_split_qi query_split_ub query_split_xu
query_split_bl query_split_fe query_split_ix query_split_mq query_split_qj query_split_uc query_split_xv
query_split_bm query_split_ff query_split_iy query_split_mr query_split_qk query_split_ud query_split_xw
query_split_bn query_split_fg query_split_iz query_split_ms query_split_ql query_split_ue query_split_xx
query_split_bo query_split_fh query_split_ja query_split_mt query_split_qm query_split_uf query_split_xy
query_split_bp query_split_fi query_split_jb query_split_mu query_split_qn query_split_ug query_split_xz
query_split_bq query_split_fj query_split_jc query_split_mv query_split_qo query_split_uh query_split_ya
query_split_br query_split_fk query_split_jd query_split_mw query_split_qp query_split_ui query_split_yb
query_split_bs query_split_fl query_split_je query_split_mx query_split_qq query_split_uj query_split_yc
query_split_bt query_split_fm query_split_jf query_split_my query_split_qr query_split_uk query_split_yd
query_split_bu query_split_fn query_split_jg query_split_mz query_split_qs query_split_ul query_split_ye
query_split_bv query_split_fo query_split_jh query_split_na query_split_qt query_split_um query_split_yf
query_split_bw query_split_fp query_split_ji query_split_nb query_split_qu query_split_un query_split_yg
query_split_bx query_split_fq query_split_jj query_split_nc query_split_qv query_split_uo query_split_yh
query_split_by query_split_fr query_split_jk query_split_nd query_split_qw query_split_up query_split_yi
query_split_bz query_split_fs query_split_jl query_split_ne query_split_qx query_split_uq query_split_yj
query_split_ca query_split_ft query_split_jm query_split_nf query_split_qy query_split_ur query_split_yk
query_split_cb query_split_fu query_split_jn query_split_ng query_split_qz query_split_us query_split_yl
query_split_cc query_split_fv query_split_jo query_split_nh query_split_ra query_split_ut query_split_ym
query_split_cd query_split_fw query_split_jp query_split_ni query_split_rb query_split_uu query_split_yn
query_split_ce query_split_fx query_split_jq query_split_nj query_split_rc query_split_uv query_split_yo
query_split_cf query_split_fy query_split_jr query_split_nk query_split_rd query_split_uw query_split_yp
query_split_cg query_split_fz query_split_js query_split_nl query_split_re query_split_ux query_split_yq
query_split_ch query_split_ga query_split_jt query_split_nm query_split_rf query_split_uy query_split_yr
query_split_ci query_split_gb query_split_ju query_split_nn query_split_rg query_split_uz query_split_ys
query_split_cj query_split_gc query_split_jv query_split_no query_split_rh query_split_va query_split_yt
query_split_ck query_split_gd query_split_jw query_split_np query_split_ri query_split_vb query_split_yu
query_split_cl query_split_ge query_split_jx query_split_nq query_split_rj query_split_vc query_split_yv
query_split_cm query_split_gf query_split_jy query_split_nr query_split_rk query_split_vd query_split_yw
query_split_cn query_split_gg query_split_jz query_split_ns query_split_rl query_split_ve query_split_yx
query_split_co query_split_gh query_split_ka query_split_nt query_split_rm query_split_vf query_split_yy
query_split_cp query_split_gi query_split_kb query_split_nu query_split_rn query_split_vg query_split_yz
query_split_cq query_split_gj query_split_kc query_split_nv query_split_ro query_split_vh query_split_za
query_split_cr query_split_gk query_split_kd query_split_nw query_split_rp query_split_vi query_split_zb
query_split_cs query_split_gl query_split_ke query_split_nx query_split_rq query_split_vj query_split_zc
query_split_ct query_split_gm query_split_kf query_split_ny query_split_rr query_split_vk query_split_zd
query_split_cu query_split_gn query_split_kg query_split_nz query_split_rs query_split_vl query_split_ze
query_split_cv query_split_go query_split_kh query_split_oa query_split_rt query_split_vm query_split_zf
query_split_cw query_split_gp query_split_ki query_split_ob query_split_ru query_split_vn query_split_zg
query_split_cx query_split_gq query_split_kj query_split_oc query_split_rv query_split_vo query_split_zh
query_split_cy query_split_gr query_split_kk query_split_od query_split_rw query_split_vp query_split_zi
query_split_cz query_split_gs query_split_kl query_split_oe query_split_rx query_split_vq query_split_zj
query_split_da query_split_gt query_split_km query_split_of query_split_ry query_split_vr query_split_zk
query_split_db query_split_gu query_split_kn query_split_og query_split_rz query_split_vs query_split_zl
query_split_dc query_split_gv query_split_ko query_split_oh query_split_sa query_split_vt query_split_zm
query_split_dd query_split_gw query_split_kp query_split_oi query_split_sb query_split_vu query_split_zn
query_split_de query_split_gx query_split_kq query_split_oj query_split_sc query_split_vv query_split_zo
query_split_df query_split_gy query_split_kr query_split_ok query_split_sd query_split_vw query_split_zp
query_split_dg query_split_gz query_split_ks query_split_ol query_split_se query_split_vx query_split_zq
query_split_dh query_split_ha query_split_kt query_split_om query_split_sf query_split_vy query_split_zr
query_split_di query_split_hb query_split_ku query_split_on query_split_sg query_split_vz query_split_zs
query_split_dj query_split_hc query_split_kv query_split_oo query_split_sh query_split_wa query_split_zt
query_split_dk query_split_hd query_split_kw query_split_op query_split_si query_split_wb query_split_zu
query_split_dl query_split_he query_split_kx query_split_oq query_split_sj query_split_wc query_split_zv
query_split_dm query_split_hf query_split_ky query_split_or query_split_sk query_split_wd query_split_zw
query_split_dn query_split_hg query_split_kz query_split_os query_split_sl query_split_we query_split_zx
query_split_do query_split_hh query_split_la query_split_ot query_split_sm query_split_wf query_split_zy
query_split_dp query_split_hi query_split_lb query_split_ou query_split_sn query_split_wg query_split_zz
query_split_dq query_split_hj query_split_lc query_split_ov query_split_so query_split_wh
query_split_dr query_split_hk query_split_ld query_split_ow query_split_sp query_split_wi
query_split_ds query_split_hl query_split_le query_split_ox query_split_sq query_split_wj

Details

	Title	Reference	Author	Source Branch	Dest Branch
	diff/import: Increase suffix length used with split	repos/mediawiki/services/ipoid!157	kharlan	main-Ia6b56aa85a1a407379497c14d7a0f3bcddddf32a	main

Customize query in GitLab

Related Objects
Search...

Status	Subtype	Assigned	Task
In Progress		Niharika	T324492 Temporary accounts - MVP
Open		None	T340895 [Epic] IP Info accommodations for temporary accounts
Open		STran	T341395 Display Spur data on IPInfo infobox
Resolved		kostajh	T339284 Deploy ipoid
Resolved	BUG REPORT	kostajh	T349056 [S] split: too many files

Event Timeline

kostajh created this task.Oct 17 2023, 6:27 AM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptOct 17 2023, 6:27 AM

kostajh added a parent task: T339284: Deploy ipoid.Oct 17 2023, 6:28 AM

kostajh changed the subtype of this task from "Task" to "Bug Report".

kostajh updated the task description. (Show Details)

Tchanders moved this task from Backlog to Deployment blockers on the iPoid-Service board.Oct 23 2023, 6:31 PM

Tchanders added a project: Trust and Safety Product Sprint.Oct 27 2023, 10:37 AM

@kostajh IIRC the problem is that this causes too much output in the logs, so we can output this many files but not print a log statement for each one.

Is that correct?

In T349056#9287159, @Tchanders wrote:

@kostajh IIRC the problem is that this causes too much output in the logs, so we can outptu this many files but not print a log statement for each one.

Is that correct?

I think it means that we need to increase the suffix length, to support the number of files that we want split to make. Per this thread we can do something like -a 5 to have a longer suffix.

In T349056#9290444, @kostajh wrote:

In T349056#9287159, @Tchanders wrote:

@kostajh IIRC the problem is that this causes too much output in the logs, so we can outptu this many files but not print a log statement for each one.

Is that correct?

I think it means that we need to increase the suffix length, to support the number of files that we want split to make. Per this thread we can do something like -a 5 to have a longer suffix.